[From scrying, mini-scrying and GPS.]
Where we have a single data source (for example, sampled white noise), as a long list of numbers (0-1024) like this:
374 427 442 425 428 436 454 438 443 462 443 444 464 442 454
in a file called “overnightwhitenoise”
which can be plotted using gnuplot from the command-line or from within GNU Emacs.
Enter at the prompt the command:
and then at the gnuplot prompt:
plot "overnightwhitenoise" with lines
with the file overnightwhitenoise in our current directory will show the plotted data on the screen
Set commands are issued before the plot command to set parameters within gnuplot. Once set we use the command “unset” eg.
- to plot to an image file/png
set term png size 1024,768 set output "overnightwhitenoise.png"
and then the plot command. We should have the png file in our current directory
- to change current directory
- to get help on commands
- to plot only part of a data set on each axis (ie. to narrow down to a certain point in time or a range of values on the y scale)
plot [0:2000] "overnightwhitenoise" with lines
to narrow to first 2000 values on the time(x) axis
plot [0:][300:400] "overnightwhitenoise" with lines
to narrow the range from 300 to 400 only on the y axis
for the title of the plot
set title "over night logging of white noise 11.03.2011"
to label the axes
set xlabel "time" set ylabel "intensity"
to label the index/data
plot "overnightwhitenoise" w lines title "white noise"
- squeezing the data by division
plot "overnightwhitenoise" using ($1/400) w lines
divides the y axis data by 400
Open a file or a buffer with gnuplot commands.
Select a region and send to/execute in gnuplot with:
C-c C-l ;; to send a line C-c C-b ;; to send the buffer
In the .org file we have:
#+begin_src gnuplot :file whitenoise.png set ylabel "intensity" set xlabel "time" plot "/root/projects/detection/logs/overnightwhitenoise" with lines #+end_src
We have data from the serial port being written as a series of numbers to a log file. In minicom we use
to log to the file in real time.
We have a gnuplot script called looper which reads that file (called test2)
a=a+1 plot "/root/test2" with lines pause 0.1 if(a<50000) reread
Opening gnuplot from the commandline we use:
a=0 load "looper"
Using code from https://github.com/gregpinero/ArduinoPlot with a few changes/additions to Arduino_Monitor.py
- to alter serial port location and arguments - to write a log of the incoming data
""" Listen to serial, return most recent numeric values Lots of help from here: http://stackoverflow.com/questions/1093598/pyserial-how-to-read-last-line-sent-from-serial-device """ from threading import Thread import time, datetime import serial # TODO: add logging to next numerical file named here last_received = '' def receiving(ser): global last_received buffer = '' while True: buffer = buffer + ser.read(ser.inWaiting()) if '\n' in buffer: lines = buffer.split('\n') # Guaranteed to have at least 2 entries last_received = lines[-2] #If the Arduino sends lots of empty lines, you'll lose the #last filled line, so you could make the above statement conditional #like so: if lines[-2]: last_received = lines[-2] buffer = lines[-1] class SerialData(object): def __init__(self, init=50): now = datetime.datetime.now() numm=now.strftime("%Y%m%d%H%M") self.f = file("%s.results.log" %numm, 'w') try: # change for SERIAL PORT LOCATION self.ser = ser = serial.Serial('/dev/ttyUSB0', 9600, timeout=1) except serial.serialutil.SerialException: #no serial connection self.ser = None else: Thread(target=receiving, args=(self.ser,)).start() def next(self): if not self.ser: return 100 #return anything so we can test when Arduino isn't connected #return a float value or try a few times until we get one for i in range(40): raw_line = last_received try: self.f.write("%s\n" %raw_line.strip()) self.f.flush() return float(raw_line.strip()) except ValueError: print 'bogus data',raw_line time.sleep(.005) return 0. def __del__(self): self.f.close() if self.ser: self.ser.close() if __name__=='__main__': s = SerialData() for i in range(500): time.sleep(.015) print s.next()
plot "allovernight" using ($1/400) w lines title "magnetic", "allovernight" using ($2/5) w lines title "light", "allovernight" w lines using 3 title "temp"
From both scry and mini-scry (with GPS attached) we have data which looks like this:
5232.07020,01325.0273,256,74,3 5232.06730,01325.0736,239,51,1 5232.06710,01325.0737,259,57,2 5232.07030,01325.0778,303,82,0
As: Latitude (NMEA WGS-84), Longitude, and, in this case, intensity of high frequency signals, low frequency, 2.4GHz
(With GSR and temperature sensor attached we have: Lat, Long, high frequency, GSR and temperature).
To plot these we need to convert NMEA to decimal using the example: 5232.0702) = 52+(32.0702/60) and convert from spherical to cartesian co-ordinates.
We can convert to decimal, then to cartesian, then plot these using gnuplot-pythin binding. Here's an example script for high and low frequency plot:
from pylab import * import csv, os, Gnuplot, Gnuplot.funcutils g = Gnuplot.Gnuplot(debug=1) NMI = 1852.0 D2R = pi/180.0 def read_csv_file(filename): data =  for row in csv.reader(open(filename)): data.append(row) return data def process_gps_data(data): latitude =  longitude =  intensity =  intensitylow =  for row in data: latitude.append(float(row[0:2]) + \ float(row[2:])/60.0) longitude.append((float(row[0:3]) + \ float(row[3:])/60.0)) intensity.append(float(row)) intensitylow.append(float(row)) return (array(latitude), array(longitude), \ array(intensity), array(intensitylow)) y=read_csv_file('/root/projects/detection/logs/scryturmcut') (lat, long, intensity,intensitylow) = process_gps_data(y) # translate spherical coordinates to Cartesian py = (lat-min(lat))*NMI*60.0 px = (long-min(long))*NMI*60.0*cos(D2R*lat) #intensitylow=intensitylow*4 newy= for x,yz,z,zz in zip(px,py,intensity,intensitylow): newy.append((x,yz,z,zz)) #gnuplot commands g('set parametric') g('set style data line') g('set surface') g('set key') g('unset contour') g('set dgrid3d 40,40,10') g('set xlabel "metres EW"') g('set ylabel "metres SN"') g('set label "signal intensity"at -60,0,0') g('set view 60,20') g.title("wasserturm 13 March 2011") g('set term png size 1024,768') #g('set term png size 14043,9933') # A0 g('set output "/root/projects/detection/logimages/scryturmcut.png"') g.splot(Gnuplot.Data(newy, using=(1,2,3), with='lines', title='high frequency'),Gnuplot.Data(newy, using=(1,2,4), with='lines', title='low frequency'))
Obviously changing any parameters such as input data file and output filename.
The result is:
- In two steps using an Emacs script and gpsbabel as follows:
Emacs Lisp to convert from scrying logfile (stripped down NMEA) to decimal (fixed 15.03.2011):
(defun nmeatodec() (interactive) (goto-char (point-min)) (while (re-search-forward "^[0-9]" nil t) ;; find lat, insert dec point, divide 60, insert (forward-char 1) (insert ".") ;; remove next . (re-search-forward "\\." nil t) (delete-char -1) ;; grab region and /60. (setq pnt (- (point) 2)) (re-search-forward "," nil t) (setq decstring (buffer-substring pnt (- (match-end 0) 1))) (setq dec (/ (string-to-number decstring) 60)) ;; kill and insert (kill-region pnt (- (match-end 0) 1)) (goto-char pnt) (insert (number-to-string dec)) (re-search-forward "," nil t) (forward-char 3) (insert ".") ;; remove next . (re-search-forward "\\." nil t) (delete-char -1) ;; same again (setq pnt (- (point) 2)) (re-search-forward "," nil t) (setq decstring (buffer-substring pnt (- (match-end 0) 1))) (setq dec (/ (string-to-number decstring) 60.0)) ;; now get rid of . (kill-region pnt (- (match-end 0) 1)) (goto-char pnt) (insert (number-to-string dec)) (re-search-backward "\\." nil t) (delete-char 1) (forward-line 1) (beginning-of-line)))
We just run
with the logfile in the buffer, re-save as “testlog” and then run:
gpsbabel -t -i unicsv -f testlog -o gpx -F testlog.gpx
- or use the conversion utility at:
- but we need first to add this to the first line of our log file:
- Convert logfile to kml for use in google earth.
gpsbabel -t -i unicsv -f testlog -o kml -F test.kml
- Use the gpsvisualizer utility to convert to kml.
(again see above for the added logfile line)
- Use the gpsvisualizer utility to overlay onto google maps.
(again see above for the added logfile line)
- Upload the kml file to a server and load on google maps
Select Layers (top level menu), new Map Layer (dropdown), Openstreetmaps(Osmarender).
Open the test.gpx file using File, Open.
Right click on the left hand text.gpx in Layer Name frame and select Properties. Here we can select Tracks and click Draw Elevation as follows:
We have to move the map to see the effect.
Right click Map from the same Layer Name frame and select Download Onscreen Maps.
The Viking screen should look something like this:
- line style 3d plot:
unset view map set ticslevel 0 set surface unset contour unset pm3d unset dgrid3d splot "/root/projects/sommercamp/120801scry.log.clean" with lines
in octave (start from commandline or GNU Emacs)
xx=load("overnightwhitenoise"); xxx=xx/1024 wavwrite("sound.wav",xxx)
Then open the file in audacity and - Effect → Normalize → remove DC offset and normalize amplitiude to -2.0dB
From the commandline:
sox test.wav -r 44100 -1 -u -b -c 1 test.dat
grep -v '^;' test.dat > test.datclean
and in gnuplot:
plot 'test.datclean' with lines every 1000
myfile = fopen("testrnd_delay_none", "r+") x =fread(myfile, "uchar"); hist(x) hist(x(1:50)) plot(x) fclose(myfile)
1] Acquire GPS works for Garmin (/dev/ttyUSB0)
2] Once acquired expand the Acquired left item and waypoints. Right click on an expanded waypoint and select GOTO
3] Maps - Add new map layer. Openstreetmap seems to work best. Make sure is moved down below our gps layer. Right click always to change options.
4] Crtl - + for zoom out and in.
5] Make a new TrackWayPoint layer and cut and paste tracks or waypoints into this and then right click to export this layer.
Viking reference: http://viking.sourceforge.net/mediawiki/index.php/Viking\_Reference\_Manual#Export
- zoom in until we can see trackpoints
- click on icon from icon bar to select edit trackpoints
- clikc on trackpoint and edit dialogue opens with:
can't run as root: remember on root: xhost +)
extract one gps track from gpx:
1] open in gpsman (running as xxxxx)
2] import gpx
3] go to data/track/export/gpx/select
1] import GPX file
2] go to items (bottom panel) → display on map → tracks
3] double click on track and delete/edit
gpsbabel -t -i garmin -f /dev/ttyUSB0 -o gpx -F gpsfile.gpx
gpsman working with tracks: http://www.words2u.net/pmwiki/?n=GPSMAN.9WorkingWithTracksBasicOperations
NMEA to decimal and googlemaps view: http://www.hiddenvision.co.uk/ez/