If you’re in the transportation field, you’ve likely heard of the Bluetooth Scanners that cost around $4,000 each. These devices scan MAC (Media Access Control) addresses and log them (with the time of the scan) and use that for travel time studies or for origin-destination studies.

My question is, can we build something good-enough with an Arduino for much less money? Something like the concept below?

There’s reasons for everything:

Arduino

Controls it all and brings it together.  Turns on the GPS, Bluetooth, listens to the stream of data from both, writes to the memory card.

GPS

The Arduino has no real-time clock (meaning that unless you tell it what time it is, it doesn’t know!).  The GPS signal includes time.  It also includes position, which would be pretty useful.

Bluetooth

If we’re going to scan for Bluetooth MAC addresses, something to receive them might come in handy…

Something to Write To

Scanning the addresses would be pretty pointless without storing the data.

Initial Design

/*
Bluetooth Tracker
Written by Andrew Rohne (arohne@oki.org)
www.oki.org
*/

#include
#include 

NewSoftSerial ol(10,11);

char inByte;
boolean ext=false;

void setup(){
  String btreturn;
  Serial.begin(115200);
  delay(1500);
  Serial.print("$$$");
  delay(1000);

}

void loop(){
  byte incomingByte=-1;
  byte index=0;
  char macaddys[160];

  while(Serial.available()>0){
    index=0;
    Serial.println("IN15");
    delay(16500);
    incomingByte=Serial.read();
    while(incomingByte>-1 && index<160){
      macaddys[index]=(char)incomingByte;
      index++;
      incomingByte=Serial.read();
    }
    if(macaddys!=""){
      Serial.end();
      writelog((String)millis()+":"+macaddys+"\r\n");
      Serial.begin(115200);
    }
  }
  if(Serial.available()<=0){
    delay(1000);
    Serial.begin(115200);
  }

}

void writelog(String line)
{
  ol.begin(9600);
  ol.print(line);
  ol.end();
}

The Results

The program wrote about 5kb of text to the file before dying after 489986 milliseconds (8 minutes). I had left it on a windowsill overnight (the windowsill is literally about 15 feet from Fort Washington Way in Cincinnati, which is 6 lanes (see below for the range centered on roughly where the setup was located).

There were 9 unique Bluetooth MAC addresses scanned. During the 8 minutes, there were 25 groups of MAC addresses written to the file. 5 MAC addresses appeared in multiple groups, with 3 of the MAC addresses appearing in 24 of the groups (and they may have appeared in the last group, it appears to have been cut off). Those same 4 have been seen in earlier tests, too, so I don't know what's going on there.

The Problems to Fix

Well, first there's the problem that I had let it run all night, and it only had 8 minutes of data. Something is causing the Arduino to stop writing or the OpenLog to stop operating.

In the output file, there are a few issues. First, some processing needs to be done, and second, it appears I am reading past the end of the serial buffer (if you look in the image below, you can see a lot of characters that look like a y with an umlaut).

In the code above, the IN15 command is sent to the Bluetooth Mate Gold, which tells it to inquire for 15 seconds, and then I delay for 16.5 seconds. This is because I THINK there is a delay after the scan finishes. I don't know how long that delay is. Vehicles traveling by at 65 MPH is 95.333 feet per second. Assuming I can get the Bluetooth device very close to the road, that 1.5 second gap SHOULD be okay, but if I have to go longer it could be a problem (the range of a Class 1 Bluetooth device is 313 feet, so a device can be scanned anytime in 626 feet (up to 313 feet before the Bluetooth Station and up to 313 feet after the Bluetooth station). A vehicle would be in range for about 6.6 seconds. However, the Bluetooth signal is at 2.4 - 2.485 Ghz, and is susceptible to some interference from the vehicle, driver, passengers, etc., so speed is key.

Conclusion

I'm on the fence as to whether or not the Bluetooth Mate Gold is the right way to do this. I will still be doing some research to see if I can get better speed out of it, or if I need to look into a different receiver that can receive the 2.4 GHz area and look for MAC addresses and stream them to the Arduino.

I also need to get the GPS up and running. That is a different story altogether, as I have been trying on that and have not been successful (despite using code that works for my personal Arduino and GPS, although the model of GPS 'chip' is different.

The script below looks at the transit line file and outputs ONLY the line code, comma-separated.  It uses a loop to check each field for ‘ NAME=’ and ‘USERN2′, which is where we now store our reporting group codes.

BEGIN{
FS=","
RS="LINE"
}
{
	for (i=1;i<20;i++)
	{
		if($i~/ NAME=/)
		{
			printf "%s,",substr($i,8,length($i)-8)
		}
		if($i~/USERN2/)
		{
			printf "%s\n",substr($i,9)
		}
	}
}

The contents of the above need to be saved to a .awk file - I used trn.awk.

To call this, I use a Pilot script to call awk and pass the input and get the output.

*awk -f {CATALOG_DIR}/INPUTS/trn.awk {CATALOG_DIR}/INPUTS/OKIROUTES.LIN >{CATALOG_DIR}/OKIROUTES.CSV

The output of this is a simple two-column comma-separated-value file of the route ID and the reporting group.

gawk /'^REPORT LINES  UserClass=Total'/,/'^Total     '/ 63PTR00A.PRN >outputfile.txt

Where 63PTR00A.PRN is the print file. Note the spaces after ^Total. For whatever reason, using the karat (the ‘^’) isn’t working to find ‘Total’ as the first thing on the line. So, I added the spaces so it gets everything. Outputfile.txt is where this will go. It will just be the table.

NOTE: You need GNUWin32 installed to do this.

One of the CTPP staff pointed me to the downloads, which are a double-edge sword. On one hand, you have a lot of data without an interface in the way. On the other hand, you have a lot of data.

I found it was easiest to use GAWK to get through the data, and it was pretty easy:

gawk '/.*COUNTY_CODE.*/' *.csv >Filename.txt

Where COUNTY_CODE is the code from Pn-Labels-xx.txt where n is the part number (1,2, or 3) and xx is the state abbreviation.

NOTE: Look up the county code EACH TIME.  It changes among parts 1, 2, and 3.

This command will go through all .csv files and output any line with the county code to the new file.

UPDATE

I have multiple counties to deal with.  There’s an easy way to start on getting a matrix:

gawk '/C4300US.*(21037|21015|21117).*32100.*/' *.csv >TotalFlowsNKY.csv

This results in a CSV table of only the total flows from three Northern Kentucky counties (21037, 21015, 21117; Campbell, Boone, and Kenton county, respectfully).  For simplicity’s sake, I didn’t include all 11 that I used.

Finishing Up

Then, I did a little Excel magic to build a matrix for all 11 counties and externals.  The formula is shown.  I have an additional sheet which is basically a cross reference of the county FIPS codes to the name abbreviations I’m using.  See the image below (click for a larger version).

After this, I built a matrix in Excel.  The matrix uses array summation (when you build this formula, you press CTRL+Enter to set it up right, else the returned value will be 0).

Using these techniques, I was able to get a journey to work matrix fairly quickly and without a lot of manual labor.

NOTE

You need to have GNUWin32 installed to use gawk.

PT (and TranPlan’s LOAD TRANSIT NETWORK, and probably TRNBUILD, too) will attempt to assign all trips.  Trips that are not assigned are output into the print file.  In PT (what this post will focus on), will output a line similar to this:


W(742): 1 Trips for I=211 to J=277, but no path for UserClass 1.


When a transit path is not found.  With a transit on-board survey, there may be a lot of these.  Therefore, less time spent writing code to parse them, the better.

To get this to a file that is easier to parse, start with your transit script, and add the following line near the top:


GLOBAL PAGEHEIGHT=32767


This removes the page headers. I had originally tried this with page headers in the print file, but it created problems. Really, you probably won’t print this anyway, so removing the page headers is probably a Godsend to you!

Then, open a command line, and type the following:

gawk '/(W\(742\).*)\./ {print $2,$5,$7}' TCPTR00A.PRN >UnassignedTransitTrips.PRN

Note that TCPTR00A.PRN is the transit assignment step print file, and UnassignedTransitTrips.PRN is the destination file. The {print $2,$5,$7} tells gawk to print the second, fifth, and seventh columns. Gawk figures out the columns itself based on spaces in the lines. The >UnassignedTransitTrips.PRN directs the output to that file, instead of listing it on the screen.

The UnassignedTransitTrips.PRN file should include something like:


1 I=3 J=285,
1 I=3 J=289,
1 I=3 J=292,
1 I=6 J=227,
1 I=7 J=1275,


The first column is the number of unassigned trips, the second column is the I zone, and the last column is the J zone.

This file can then be brought into two Matrix steps to move it to a matrix. The first step should include the following code:

RUN PGM=MATRIX PRNFILE="S:\USER\ROHNE\PROJECTS\TRANSIT OB SURVEY\TRAVELMODEL\MODEL\TCMAT00A.PRN"
FILEO RECO[1] = "S:\User\Rohne\Projects\Transit OB Survey\TravelModel\Model\Outputs\UnassignedAM.DBF",
 FIELDS=IZ,JZ,V
FILEI RECI = "S:\User\Rohne\Projects\Transit OB Survey\TravelModel\Model\UnassignedTransitTrips.PRN"

RO.V=RECI.NFIELD[1]
RO.IZ=SUBSTR(RECI.CFIELD[2],3,STRLEN(RECI.CFIELD[2])-2)
RO.JZ=SUBSTR(RECI.CFIELD[3],3,STRLEN(RECI.CFIELD[3])-2)
WRITE RECO=1

ENDRUN

This first step parses the I=, J=, and comma out of the file and inserts the I, J, and number of trips into a DBF file. This is naturally sorted by I then J because of the way PT works and because I am only using one user class in this case.

The second Matrix step is below:

RUN PGM=MATRIX
FILEO MATO[1] = "S:\User\Rohne\Projects\Transit OB Survey\TravelModel\Model\Outputs\UnassignedAM.MAT" MO=1
FILEI MATI[1] = "S:\User\Rohne\Projects\Transit OB Survey\TravelModel\Model\Outputs\UnassignedAM.DBF" PATTERN=IJM:V FIELDS=IZ,JZ,0,V

PAR ZONES=2425

MW[1]=MI.1.1
ENDRUN

This step simply reads the DBF file and puts it into a matrix.

At this point, you can easily draw desire lines to show the unassigned survey trips. Hopefully it looks better than mine!

I didn’t want to write any sort of program to import the data into Access or FoxPro, and I didn’t want to mess with it (since it would be big) in Excel or Notepad++.

So, I took the first of the 244 files and named it CountData.csv. The remaining files all begin with ‘fifteen_min’ and they are isolated in their own folder with no subfolders.

Enter Windows PowerShell really powered up with GNUWin.

One command:
awk 'NR==2,NR<2' .\f*.csv >> CountData.csv

awk is a data extraction and reporting tool that uses a data-driven scripting language consisting of a set of actions to be taken against textual data (either in files or data streams) for the purpose of producing formatted reports (source: Wikipedia).

The first argument, NR==2 means start on record #2, or the second line in the file.
The second argument, NR<2, means end on the record less than 2. In this case, it always returns false, and thus the remainder of the file is output. The .\f*.csv means any file in this folder where the first letter is f and the last 4 letters are .csv (and anything goes between them). The ‘>> CountData.csv’ means to append to CountData.csv

Once I started this process, it ran for a good 45 minutes and created a really big file (about 420 MB).

After all this, I saw a bunch of “NUL” characters in Notepad++, roughly one every-other-letter, and it looked like the data was there (just separated by “NUL” characters).  I had to find and replace “\x00″ with blank (searching as Regular Expression).  That took a while.

Acknowledgements:

The Linux Commando.  His post ultimately helped me put two and two together to do what I needed to do.

Security 102.  The NUL thing.

The first step is to go to the options menu by clicking on the wrench icon and going to Options:

The second step is to go to the Basics tab (or on Mac, click on the Basics icon)

The third step is to add the search engine.  On Windows, click Add, and then fill out the resulting form, on OS X, click the ‘+’ button and do the same.

Windows Form:

The following are the items for the form:

Name: UPS

Keyword: UPS

URL: http://wwwapps.ups.com/WebTracking/processInputRequest?sort_by=status&tracknums_displ ayed=1&TypeOfInquiryNumber=T&loc=en_US&InquiryNumber1=%s&track.x=0&track.y=0

NOTE: The entire URL above should be one line with no spaces!

Click OK on everything (or in some cases, the red circle on OS X).  To use this, open Chrome, type ‘ups’ in the address bar and press Tab and enter the tracking number (copy-paste works well for this).

Once you press Enter, you will immediately go to the UPS website showing your tracking information.  In this case, my shipment won’t make it by Christmas.  Oh well.

The first thing that is needed is a python script.  The below is an example that we’re not using at the moment, but it merges multiple transit line files together.

import arcgisscripting, sys, os
gp=arcgisscripting.create()

gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolBox/Toolboxes/Data Management Tools.tbx")

print sys.argv

input1=sys.argv[1]
input2=sys.argv[2]
output=sys.argv[3]

in1=input1+input1[input1.rfind("\\"):]+"_PTLine"
in2=input2+input2[input2.rfind("\\"):]+"_PTLine"

input=in1+';'+in2
input=input.replace("\\","/")
output=output.replace("\\","/")

print input
print output

if gp.Exists(output):
    gp.delete_management(output)

#input=input1+"_PTLine" +';'+input2 +"_PTLine"

gp.Merge_management(input,output)

print gp.GetMessage

del gp

To call this, we add the following in a Pilot script:

*merge.py {CATALOG_DIR}\Inputs\Transit.mdb\Routes1 {CATALOG_DIR}\Inputs\Transit.mdb\Routes2 {CATALOG_DIR}\Inputs\Transit.mdb\RoutesCombined

This makes it easy to create geoprocessing steps in ArcMap, export them to Python, and call them from the model.

Over the years, I’ve amassed a set of utilities, many of them free, to make my life easier.  This list can sometimes take the place of things that I would normally use in the office, other times they are things that sync to the “cloud” and I use them both in the office and at home.

1. Dropbox

I don’t care too much for USB “thumb” drives, and I’ve had my fair share of leaving them at home or at work and needing them at the opposite location.  Dropbox clears up this mess, as there are no USB drives to lose or leave in the wrong place.  NOTE: the link that I have IS a referral link.  Clicking on that and creating an account results in both of us getting an extra 250 MB of space with the free account (starts at 2 GB, max for free is 8 GB).

2. Evernote

I take a lot of notes, both on the road at conferences and at the office.  Evernote is what I use to keep them organized.

3. Google Docs

Unless you want to spring for Microsoft Office at home, Google Docs is the way to go.  There are several others including Zoho and Office Online, but I haven’t used them.  Google Docs has great collaboration features, document versioning, and its free.  Just make sure to back it up! The only problem: no DBF file support.

4. Notepad++

This is perhaps the greatest text editor.  It understands and does some context highlighting (etc) for many programming languages.  Even better, Colby from Citilabs uploaded his language definition file for Cube Voyager to the user group!

5. Microsoft Visual {whatever} Express Edition

The Express Edition tools have become our go-to tools for new development, particularly MS Visual C++ EE and MS Visual Basic EE.  Since they’re free, you can have copies both at home and work.

6. Eclipse

This one’s almost optional, but for those working with Java models, this is the standard IDE, and it is open source.

Any tools to add?  Add them in the comments below.

In my case, I have a control file for my nested logit mode choice program, and because of the age of the mode choice program, I want to redesign it.  The first part of this is reading the control file, and I did a little trick to help with reading each control file line.  With C++, there is no way to read variables in from a file (like there is with FORTRAN).

The first part of the code reads the control file, and you will see that once I open and read the control file, I section it out (the control file has sections for files ($FILES), operation parameters ($PARAMS), operation options ($OPTIONS), and mode choice parameters ($PARMS). Each section ends with an end tag ($END). This adds the flexibility of being able to re-use variables in different locations.

After the section, the next portion of the code reads the line and checks to see if FPERIN is found. If it is, a ControlFileEntry object is created. This object is a class that is used to return the filename held in the object. This makes it easy to reduce code.

int readControlFile(char *controlFileName){
	cout << "Reading " << controlFileName << endl;
	//Read the control file
	string line;
	bool inFiles=false, inParams=false, inOptions=false, inParms=false;
	ifstream controlFile(controlFileName);
	if(!controlFile.good()){
		cout << "PROBLEMS READING CONTROL FILE" << endl;
		return 1;
	}
	while(controlFile.good()){
		getline(controlFile,line);
		//check the vars sections
		if(line.find("$FILES")!=string::npos)
			inFiles=true;
		if(line.find("$PARAMS")!=string::npos)
			inParams=true;
		if(line.find("$OPTIONS")!=string::npos)
			inOptions=true;
		if(line.find("$PARMS")!=string::npos)
			inParms=true;
		if(line.find("$END")!=string::npos){
			inFiles=false;
			inParams=false;
			inOptions=false;
			inParms=false;
		}
		if(inFiles){
			cout << "Checking files" << endl;
			if(line.find("FPERIN")!=string::npos){
				controlFileEntry cfe(line);
				PerTrpIn=cfe.filename;
			}
//SNIP!!!
	return 0;
}

The controlFileEntry is code is below.  This is used at the top of the code, just below the preprocessor directives (the #include stuff).

class controlFileEntry{
public:
	string filename;
	controlFileEntry(string Entry){
		beg=(int)Entry.find("=")+2;
		end=(int)Entry.rfind("\'")-beg;
		filename=Entry.substr(beg,end);
	}
	~controlFileEntry(){
		beg=0;
		end=0;
		filename="";
	}
private:
	string Entry;
	int beg;
	int end;
};

The class has one public member, filename, which is what is read in the code where it is used. There are two public functions. The first is the constructor (controlFileEntry) which is used when creating the object. The second is the de-constructor (~controlFileEntry), which sets the beg, end, and filename variables to zero and blank.  The beg, end (misnomer), and the line sent to it are private and cannot be used in code.

This can be extended, as the file entry type is fine when there are quotes around the item (it is setup for that, note the -2 in beg).  I wrote a different one for integers, floating point, and boolean values.

class controlParamEntry{
public:
	int ivalue;
	bool bvalue;
	double dvalue;
	controlParamEntry(string Entry){
		beg=(int)Entry.find("=")+1;
		end=(int)Entry.rfind(",")-beg;
		ivalue=0;
		dvalue=0;
		if(Entry.substr(beg,end)=="T"){
			bvalue=true;
			ivalue=1;
			dvalue=1;
		}else if(Entry.substr(beg,end)=="F"){
			bvalue=false;
			ivalue=0;
			dvalue=0;
		}
		if(ivalue==0){
			ivalue=atoi(Entry.substr(beg,end).c_str());
		}
		if(dvalue==0){
			dvalue=atof(Entry.substr(beg,end).c_str());
		}
	}
	~controlParamEntry(){
		beg=0;
		end=0;
		ivalue=0;
		dvalue=0;
		Entry="";
	}
private:
	string Entry;
	int beg;
	int end;
};

As you can see above, there are return values for floating point (dvalue), integer (ivalue), and boolean (bvalue).

Tune in next week to see more of the code.

Setup

Like last week, we need to reference VoyagerFileAccess.lib in the project.  We also need to add the external references in the header file as below:

extern "C" __declspec(dllimport)void* MatReaderOpen(const char *filename, char *errMsg, int errBufLen);
extern "C" __declspec(dllimport)int MatReaderGetNumMats(void* state);
extern "C" __declspec(dllimport)int MatReaderGetNumZones(void* state);
extern "C" __declspec(dllimport)int MatReaderGetMatrixNames(void* state, char **names);
extern "C" __declspec(dllimport)int MatReaderGetRow(void* state, const int mat, const int row, double *buffer);
extern "C" __declspec(dllimport)void MatReaderClose(void* state);

Also ensure that the project is setup with the character set of “Not Set” as opposed to Unicode, which seems to be a default in MS Visual C++ Express Edition.

Main Application

The main application is fairly simple and just opens the matrix and outputs the number of tables and zones to the screen.

#include "stdafx.h"
#include <stdio.h>
#include <iostream.h>

using namespace std;

int _tmain(int argc, _TCHAR* argv[])
{
	char errMsg[256]="";
	// Open the matrix
	void* matrixState=MatReaderOpen(argv[1],errMsg,256);
	// Get number of tables in the matrix
	int nMats = MatReaderGetNumMats(matrixState);
	// Get number of zones in the matrix
	int nZones = MatReaderGetNumZones(matrixState);
	// Output to screen
	cout << "File " << argv[1] << endl;
	cout << "Number of Tables....." << nMats << endl;
	cout << "Number of Zones......" << nZones << endl;
	// Close the matrix
	MatReaderClose(matrixState);
	cout << "Matrix Closed." << endl;
	cout << "Press any key to close" << endl; 	char tmp; 	cin >> tmp;
	return 0;
}

The output looks like the below:

The first thing with the path files is that it is (as indicated in the documentation) Highway Path Files only.  I first tried PT Route Files (which can be read in Viper in the same way one could use Highway Path files), but alas, you receive an error when trying to do that.

For this, I have created a console application, which could become something to run in a model run.

Setup

The first thing is to setup your include file with the DLL references.

Start by adding a reference to VoyagerFileAccess.lib.  In Visual C++ Express 2010, right-click on your solution name and add an existing item (and point it to VoyagerFileAccess.lib).  Then, in a header file (or in your source file, but normal programming conventions seem to dictate that these items belong in headers), add the following lines:

extern "C" __declspec(dllimport)void* PathReaderOpen(const char *filename, char *errMsg, int errBufLen);
extern "C" __declspec(dllimport)void* PathReaderClose(void* state);
extern "C" __declspec(dllimport)int PathReaderGetNumZones(void* state);
extern "C" __declspec(dllimport)int PathReaderGetNumTables(void* state);

These lines tell the compiler that these four functions are imported through a DLL (and thus, it uses the lib file to know where to go).

The next thing, since this is a console application, is to correct the Character Set.  Right-click on the solution, go to properties, select Configuration Properties – General, and set the Character Set to “Not Set”.  If you leave it on Unicode, your command line arguments will have only one letter.  See the screen capture below.

Main Application

This is a small application that just shows some simple reading the zones and tables in the path file.  The application takes one command-line argument.

The source, fully commented, is below.

#include "stdafx.h"

#include <Windows.h>

#include <stdio.h>

#include <iostream>

using namespace std;

int _tmain(int argc, char* argv[])

{

// dim variables
char errorMessage[256]="";
int Zones,Tables;

// Opens the path file and sets the Zones and Tables variables
void* state=PathReaderOpen(argv[1],errorMessage,256);
Zones=PathReaderGetNumZones(state);
Tables=PathReaderGetNumTables(state);

// Dumps the variables to the screen
cout << "State of PathReaderOpen: " << state << endl;
cout << "PathReaderErrorMessage: " << errorMessage << endl;
cout << "Zones: " << Zones << endl;
cout << "Tables: " << Tables << endl;

// Closes the path file
PathReaderClose(state);
cout << "Path Reader Closed";

// This makes the command window wait for input from the user before closing
char tmp;
cin >> tmp;

return 0;
}

For debugging, you will want to set the command-line argument.  This is done by right-clicking on the solution and going to Configuration – Debugging.  See the screen capture below.

Output

The output of this is fairly simple:

In the coming weeks, I will post more about using this new API.

Building on last weeks post, the below shows the modifications necessary in Cube. The first thing I added is the variable itself (else, you will get one of those inexplicable errors). In this case, I add MDARRAY2 as a variable that is the same dimensions as MDARRAY. The second part that I add (which is after the CALL statement) is just to report the values stored in MDARRAY2.

RUN PGM=MATRIX PRNFILE="C:\TEMP\DTMAT00B.PRN"
FILEO PRINTO[1] = "C:\TEMP\DEBUG.PRN"

PAR ZONES=1

ARRAY MDARRAY=5,5, MDARRAY2=5,5

LOOP _c=1,5
  LOOP _r=1,5
    MDARRAY[_c][_r]=RAND()
    PRINT PRINTO=1 LIST='MDARRAY[',_c(1.0),'][',_r(1.0),']=',MDARRAY[_c][_r](8.6)
  ENDLOOP
ENDLOOP

CALL DLL=DLLFILE(TableReader)

LOOP _c=1,5
  LOOP _r=1,5
    PRINT PRINTO=1 LIST='MDARRAY2[',_c(1.0),'][',_r(1.0),']=',MDARRAY2[_c][_r](8.6)
  ENDLOOP
ENDLOOP
ENDRUN

In C++, I add a second variable for MDARRAY2 (called TableRecord2). It is critical that this is a double* variable, as this needs to be a pointer so Cube can access updated values of the variable. Similar with how I read MDARRAY into TableRecord, I do the same with MDARRAY2 and TableRecord2, which reads the pointers to MDARRAY2 into TableRecord2. Then, as I iterate through TableRecord, I set TableRecord2 to 10 * TableRecord. After this, the DLL is complete and Cube ultimately prints all the values to the print output.

int TableReader (Callstack* Stack){
	double* TableRecord;
	double* TableRecord2;
	char message[100];

	TableRecord=(double*)Stack->pfFindVar("MDARRAY",0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,
                16,17,18,19,20,21,22,23,24);
	TableRecord2=(double*)Stack->pfFindVar("MDARRAY2",0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,
                16,17,18,19,20,21,22,23,24);
	for(int x=0;x<=24;x++){ 	if(&TableRecord!=0){ 			 		sprintf(message,"TableRecord=%f",TableRecord[x]); 		Stack->pfPrnLine(1,message);
		TableRecord2[x]=TableRecord[x]*10;
		}
	}
	return 0;
}

Additional Considerations

If you decide to use this, you may want to pass the sizes of each dimension if it is important. Then, you can write a function to take the sequential value and return the column or row.

One of the cool things with the latest version of Cube Voyager is multi-dimensional arrays. However, it appears behind the scenes (or at least to C++) that the multi-dimensional arrays are a wrapper over a single-dimension array.

The easiest way to show this is to make a random array and send it to the print file. Making the random array in Cube is simple:

RUN PGM=MATRIX PRNFILE="C:\TEMP\DTMAT00B.PRN"
FILEO PRINTO[1] = "C:\TEMP\DEBUG.PRN"

PAR ZONES=1

ARRAY MDARRAY=5,5

LOOP _c=1,5
  LOOP _r=1,5
    MDARRAY[_c][_r]=RAND()
    PRINT PRINTO=1 LIST='MDARRAY[',_c(1.0),'][',_r(1.0),']=',MDARRAY[_c][_r](8.6)
  ENDLOOP
ENDLOOP

CALL DLL=DLLFILE(TableReader)
ENDRUN

Then, in C++ we can pull 25 (!) array values from this:

int TableReader (Callstack* Stack){
	double* TableRecord;
	char message[100];

	TableRecord=(double*)Stack->pfFindVar("MDARRAY",0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,
             16,17,18,19,20,21,22,23,24);
	for(int x=0;x<=24;x++){
	if(&TableRecord!=0){
		sprintf(message,"TableRecord=%f",TableRecord[x]);
		Stack->pfPrnLine(1,message);
		}
	}
	return 0;
}

For fixed size multi-dimensional arrays, this isn’t really an issue. It would be very easy to wrap the Stack->pfFindVar line in a set of loops that fills a multi-dimensional array.

Fixing this requires a few minor modifications to the source in {download}\src\googlecl\docs\base.py (where {download} is where you downloaded the files}.

The first fix is in the try block on line 51.
was:

from gdata.docs.data import DOCUMENT_LABEL, SPREADSHEET_LABEL, \
                              PRESENTATION_LABEL, FOLDER_LABEL, PDF_LABEL

To:

from gdata.docs.data import DOCUMENT_LABEL, SPREADSHEET_LABEL, \
                              PRESENTATION_LABEL, FOLDER_LABEL, PDF_LABEL, DRAWING_LABEL

Then, beginning on 52 (the except ImportError block), it should include DRAWING_LABEL = ‘drawing’ as below:

except ImportError:
  DOCUMENT_LABEL = 'document'
  SPREADSHEET_LABEL = 'spreadsheet'
  PRESENTATION_LABEL = 'presentation'
  DRAWING_LABEL = 'drawing'
  FOLDER_LABEL = 'folder'
  PDF_LABEL = 'pdf'

Then, on line371, the following needs to be added before the ‘else’:
except ImportError:

elif doctype_label == DRAWING_LABEL:
      return googlecl.CONFIG.get(SECTION_HEADER, 'drawing_format')

Finally, in your .googlecl file (mine is under my “profile drive” because of our network settings, your mileage likely will vary, so you’ll have to search for it), open config in any text editor and add the following in the [DOCS] section:

drawing_format = png

Note: while you’re at it, you might want to change document_format = txt to document_format = doc

That’s it. Now if you run ‘google docs get .* ./backup’, you get the drawings as well.

Extending the posts relating to using C++ with Cube Voyager, the next thing to look at is using C++ with the DBI input module that was added in Cube 5.1.

The key to making this happen is the Matrix FILEI help section that discusses that certain things are held in arrays. My last post on this subject got into arrays a little, but there are a few new tricks to use.

The code below (C++) is some simple source code that reads the input database (DBI.1) and reads the built-in array of field values.

typedef struct { int I,J,Zones,Iterations;
				double** MW;
				void* (*pfFindVar)(char*,...);
				void* (*pfPrnLine)(int,char*);
} Callstack;

int TableReader (Callstack* Stack){

	double* TableRecord;
	char message[100];

	TableRecord=(double*)Stack->pfFindVar("DBI.1.NFIELD",1,2,3,4,5,6,7,8,9,10,
		11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,
		31,32,33,34,35,36,37,38,39,40);

	for(int x=0;x<=40;x++){ 		if(&TableRecord[x]!=0){ 			sprintf(message,"Table %i=%f",x,TableRecord[x]); 			Stack->pfPrnLine(1,message);
		}
	}

	return 0;
}

This reads all the NUMERIC (note the emphasis!) fields and dumps them to the print file. There is a trick in here – I have a table with 39 fields, but I pull 40 fields. If you actually use that 40th field in the sprintf statement, it crashes. This is why I test to ensure that &TableRecord[x] (the pointer to the table record) does not equal 0.

Normally in Cube, one would read the database fields using DI.1.FIELDNAME. This is certainly possible in C++. Note the code below (where HHPERSONID is the field name):

int TableReader2 (Callstack* Stack){
	double HHPersonId;
	char message[100];

	HHPersonId=*(double*)Stack->pfFindVar("DI.1.HHPERSONID");
	sprintf(message,"%f",HHPersonId);
	Stack->pfPrnLine(1,message);

	return 0;
}

This is similar to the code example above.

Tune in next week when I get into more DBI with C++.

RTFM (Read The Fine Manual)

Read the help file for Matrix CALL statement.  The struct statement is pretty important, and the sprintf lines will be used throughout.

Memory Pointers

One of the most important things to understand is that because there are so many variables that can be passed between Cube and the C++ DLL, the memory pointers are passed instead.  Also, one of those “pull your hair out” things relates to this – if you attempt to access a memory pointer that hasn’t been initialized, you get a crash that gives no error.

Because of this, the variables in the struct statement have a *, which notes that it is a memory pointer.

To keep from getting the crash-with-no-error, the following statement works well to test and allows a default to be used if the variable ‘MarketSegments’ is not set in Cube.

int MarketSegments=4;

if(Stack->pfFindVar("MarketSegments")!=0)
MarketSegments=(int)*Stack->pfFindVar("MarketSegments");

Matrix In, Matrix Out

While the Help file says that you can get to defined working matrixes using

static double **MW;
MW=(*Stack->pfFindVar)("MW",1);

I can’t get it to work using C++ (I have gotten it to work in C).  Instead, use the following:

static double **MW=NULL;
MW=Stack->MW;

This will enable you to use MW[m][j] (where m is the MW number, and j is the j-zone).

You can also set the MW variables, but it does NOTHING if you don’t set the MW to something in Cube Voyager.  Ergo, if you set

MW[101][j]=10;

Your output will be 0 unless you do the following in Cube Voyager

MW[101]=0
CALL...

Array Variables

One of the tricks I use to get array variables out of Cube is this

float ArrayVariable[7]={0,0,0,0,0,0,0};  //Note: I'm using 1-6.  Setting this to 7 means 0-6.  Setting it to 6 would mean 0-5
if(Stack->pfFindVar("ArrayVariable")>0){
double* tmpAV=NULL;
tmpAV=Stack->pfFindVar("ArrayVariable",1,2,3,4,5,6);
for(int x=1;x<=6;x++)
ArrayVariable[x]=tmpAV[x];
}

This code above checks that the ArrayVariable, fills them into a temporary variable, and then sets the actual variable.

Compilation Linker Settings

When compiling, you need to set the EXPORT flag so the name is predictable and correct.  To do this, go to your project’s property pages – Configuration Properties – Linker – Command Line.  You need to add “/EXPORT:FunctionName” under Additional Options.  See the screenshot below

.

Other Weird Stuff

Any error in C++ that does not cause a compilation error results in one of those useless “this program has an error and will be closed” and crashes Task Monitor.  That being said, write messages to the output file frequently (if at least during debugging).  This can assist with finding typos (like, say, %10.65f in an sprintf statement, which means 65 decimal places in a 10-width line).

In Matrix when using DBI, PAR ZONES=1 will effectively shut off Cluster. Therefore, the following works really well.


DISTRIBUTEINTRASTEP ProcessID=Cluster ProcessList={CORES}

LOOP _r=start,end
x=DBIReadRecord(1,_r)
ENDLOOP


This script sets each core to process an equal portion of the database with any remainder (e.g if you cluster 4 records over 3 cores) to the last core.

In Cube Voyager, I’ve been finding more ways to cut runtimes down as much as possible, and below is the list with some of the rationale.

Keep JLoops at a Minimum

The Matrix program runs in an implied ILOOP.  This being the case, anything in the script runs many times (as many as the zones you have).  Using a JLOOP in a 2,000 zone model means that there are  4,000,000 calculations to be done for each COMP statement.  What happens if you have 3 COMP statements?  You go from 4,000,000 to 12,000,000 calculations.  This is even worse if the calculations include a lookup function, which are generally slow.

Keep Files Small – Only Write What You Have To

This is a no-brainer.  The more that Cube (or anything, for that matter) has to write to disk, the longer the runtime.

Replace RECI with DBI wherever possible

DBI can be clustered (look for a post on that in the coming weeks).  While I’m not sure if there is any difference on one core, being able to use Cluster is the trump card.

Use Cluster Dynamically and Wherever Possible

Standardize the Cluster ID in your code, but set the process list to a catalog key as with below:

DISTRIBUTEINTRASTEP CLUSTERID=Cluster PROCESSLIST=2-{CORES}

Using Cluster to your advantage is critical to having a fast model.

The first step to this is to have a Blackberry.  I’m sure there is a way to do this with an Android based phone or an iPhone, but I haven’t used them, so I don’t know how to do this on those.

On your Blackberry, open App World and search for, download, and install GPSed.  This program can be used to track your location.  BIG IMPORTANT NOTE: this will use much more battery than normal.  The amount of battery is used WAY MORE if you go into an area with no service.

Before you go ‘Shooting’

First off, set the time on your camera.  It will make your life easier down the road.

Then, at the start of your shoot/photowalk, open GPSed and select New Track… in the menu:

->

And give it a name:

Start shooting!  While you are shooting, the GPSed free version log a GPS point roughly every 2 seconds.  This can be affected by a number of things,like tall buildings, clouds, trees, and having your Blackberry in your pocket or a case.  While those do create problems, leaving my Blackberry in its case while I walk through downtown is pretty good, but will sometimes miss points.

When you are done…

When you are done, open the menu ans select Finish Track.  On the following screen, it will ask you if you want to share or upload the track.  I tend to use “Do Nothing”.

->

After the track is saved (and uploaded or shared if you did that), you will need to convert the track to GPX format.

The next steps on the Blackberry can be done now or at home.  I tend to do this part at home, since it goes hand-in-hand with the rest of the process.

At Home…

In GPSed, click the menu and select Pages > Track List.  You will want to select your track, click the menu, and select Convert to GPX…

-> ->

->

It will take a few to process.  My ~2 hour, 3 mile photowalk around 15-30 seconds.  Make sure you remember where it tells you it put the file!

->

After this, you can close GPSed.  You will want to locate that file and transfer it to your computer.  I used email.

-> ->

On Your Computer…

At this point, all of the Blackberry steps are completed.  You should be able to open your email and get the GPX file to process in Lightroom.  Save this somewhere.  I use /Users/andrew/Pictures/Geoencoding on my Mac.

For this part, you will need Jeffrey Friedl’s GPS Plugin.  If you have a lot of pictures, you will want to donate to him to remove the 10-pic-at-a-time block.  Also, if you do this more than once or twice in your life, you should donate to him

To set the GPS coordinates on the actual pictures, open Lightroom and import your pictures.  Also, use the Lightroom Plugin Manager to open the GPS plugin.

Select all the images you want to set GPS coordinates to (CMD-A on Mac, Cntrl-A on PC, if you have them in the same folder).  Then, go to File-Plugin Extras – Geoencode…

There are several important things on the resulting window…

The important stuff here:

• Make sure the tracklog tab is selected.
• Make sure you’ve selected the GPX file that you emailed to yourself.
• Select UTC as the timezone – this is because the times from the GPS satellites are in UTC.
• I’ve had the most luck with a 30 second fuzzyness.  Increase if you are in a downtown, decrease if you are not and you were running.
• If your camera doesn’t have the correct time, correct it!  You really shouldn’t have to use much in the camera time correction.

Once you click Geoencode, the GPS information will be added to all the pictures it can.  Congratulations, your pictures are mappable!  Now, if you use Jeffery Friedl’s Flicker plugin, Flickr will know where to place them on the map!

No.

Fifty years ago, when the trip based modeling methodologies were developed, the primary reason was to evaluate highway improvements.  While tolling was in use, the bonding requirements were likely different.  Transit, while extremely important, was not in the public realm (the streetcars were normally privately owned by the area’s electric company).

Now, there are a lot of demands on travel models:

• Tolling/Toll Road analysis at a better level
• Different tolling schemes (area tolling, cordon tolling)
• Travel Demand Management (telecommuting, flex hours, flex time, alternative schedules)
• Better freight modeling (which now is becoming commodity flow and commercial vehicle modeling)
• Varying levels of transit (local bus, express bus, intercity bus, BRT, light rail, and commuter rail

While many of these can be done with trip based models, most of them cannot be done well with trip based models.  There are a number of reasons, but the few that come to mind are aggregation bias, modal inconsistency, and household interrelationships.

Aggregation Bias

Aggregation bias occurs when averages are used to determine an outcome.  For example, using a zonal average vehicles per household, you miss the components that form the average, such as:

20 households, average VPHH = 2.2
2 HH VPHH = 0
5 HH VPHH = 1
4 HH VPHH = 2
6 HH VPHH = 3
3 HH VPHH = 4+

The trip generation and modal choices (car, bus, bike, walk, etc.) among these households are all different, and are even more more different if you look at the number of workers per household.

Modal Inconsistency

In trip based modeling, “people” are not tracked throughout their day.  So, if someone rides the bus to work, there is nothing in the model to ensure that they don’t drive from work to get lunch.  While we don’t want to force people to use the same mode, since many people will use the bus to get to work and then walk to lunch or to go shopping during lunch, we want to make sure that there is some compatibility of modes.

Household Interrelationships

One of the features of of tour based models is determining each person’s daily activity pattern.  During this process, certain person types can determine what another person is doing.  For example, if a preschool age child is staying home, an adult (whether they are a worker or not) HAS to stay home.  Another example is if a school-non-driving-age child is going on a non-mandatory trip, an adult must accompany them.  Trip based models don’t know about the household makeup and the household interaction.

The above are only three of the many reasons why tour-based modeling is important.  There are many more, but I feel these are some of the most important and some of the easiest to understand.

I was disappointed.

Three things stuck out in my mind as being flat-out wrong.  The first was “The best case scenario is when you (the consultant) write the scope for the RFQ”.  The second was “The best way is sole source contracts”.  The third was constantly using RFP as a tool to limit the responses from consultants to only those that you want to respond.

Consultants Writing Scopes for RFQs

Looking at the AICP Code of Ethics, it seems that if a consultant writes the scope for the RFQ (or RFP), I feel it is in violation of Part A, 2a and 2c.  If a consultant is writing the scope for me, where is my professional judgement?  Does that judgement not extend to what I feel my needs (and my organizations needs) are?  Both of those are brought up in 2a.  Looking at 2c, which is avoiding a conflict of interest, it seems to me that if a consultant writes the scope for an RFQ, that is a direct conflict of interest – the consultant is going to write the scope that gives them an advantage (whether intentionally or unintentionally).

Sole Source?

When being audited by the State of Ohio Auditors, you are under extreme scrutiny when trying to sole-source a contract.  The reasons why are obvious.  A few years ago, my department attempted to sole-source a contract because it was a $30k contract and it seemed that there was only one firm that could do the job for that price.  While that may have been correct, there was several firms willing to try.  The job ultimately went to a firm that was NOT going to be the one that would have received the sole-source contract (there is a lot of talk that they may have taken a loss on the job, but I would venture a guess that the others would have as well).  Had the sole-source been allowed to continue, it would have been considered illegal under Ohio law and my organization would have been fined.

I can’t type all this without bringing up another big issue that CAN negate the above.  General Planning Consultants and General Engineering Consultants.  The GPC and GEC contracts are always put up for RFQ, and handing a scope to a GEC or GPC consultant is NOT the same as sole-source.  This method is perfectly legal (it is open to public review and open to all consultants to submit statements of qualifications) and is a great way to get smaller (less than $100k, perhaps) jobs to consultants without them spending a lot of money trying to get smaller jobs.  They have to spend their marketing money up-front, but over the 3-5 year span, they have plenty of opportunity to make it back on smaller jobs that have very small marketing requirements.

RFPs Only to Certain Consultants?

Again, 2c, conflict of interest – public agencies cannot perform the work of the public good using the fewest tax dollars without having an open bid process.  Also, it is pretty likely that every state requires RFPs and RFQs to be advertised.  That being said, what’s the point?  You’re going to send the RFP to 2 or 3 consultants but post it on your website (and for us, the newspaper, state DOT website, and various other locations as required by law and our policy) for all to see?  Sounds like a pretty ineffective way to only target a few consultants.

If you only want certain consultants to respond, find a way to do it, legally, without giving the opportunity for other consultants to not compete for it.

The big difference between these two scenarios was intention.  Earlier in the race at AMS, Kesolowski got into the side of Edwards, causing Edwards a long repair and a poor finish.

NASCAR handed down a three-race probation to Edwards after parking him for the remainder of the race at AMS.  The debate as to whether that was the most appropriate disciplinary action have been swirling around NASCAR for weeks (and still is at the time of writing).

This post is not about whether NASCAR made the right or wrong decision, but rather how it relates to management.

You have to understand the history behind the wing.  If you’ve watched the videos above, you’ve seen two of three.  The other piece of history is at this video.  The scenario at AMS is the third time that a car has become airborne after being turned around.

The probation that Edwards faces (and no suspension or fine, mind you) was because Edward’s intent was to mess up Kesolowski’s 6th place finish with a spin to the infield.  Edwards didn’t intend for the vehicle to flip, and the vehicle should not have flipped.  In fact, the severe crash was likely caused more by the wing on the back of the car (which has now been replaced with a spoiler), not by Edwards’s intentionally spinning Kesolowski.

This is quite a conundrum for NASCAR.  They control the design of the car very strictly.  They also said that the drivers could use a little less restraint after feeling a lot of criticism over the 2009 season where they made rules that limited the drivers actions.  Drivers and teams are not allowed to make decisions as to whether they use the wing or not.  They have to use it.

The important thing here is, as a manager, make the decision looking at all pieces of information and all parts of history.  Look at what you’ve told your employees.  Look at what has happened in the past that your employees should have been aware of.  Look at what you would have done in that situation, particularly if you weren’t a manager.  Discuss the issue with the employees involved.  Do not make rash decisions and do not let emotions be the only thing that guides your decisions.

The long answer (and previously blogged): the study was a before and after study that claimed that cell phone laws have no effect.  Such is the case when you don’t and can’t enforce it.  Cell phone bans are difficult to enforce (as are speed limits).  Most police departments have their hands full as it is, and their budgets are being affected (negatively) by the economy.  The original study was basically a before and after study of the same conditions on both sides.

Why Cell Phones and Driving Aren’t Dangerous | Psychology Today.

While a little odd, I applaud the mayor for experimenting with a low-cost, low-impact way to handle the problem.  I hope it works.

Romanian street sign warns drivers of ‘drunk pedestrians’ – Telegraph.

• For the $400 I would have spend on APA+AICP, I can have ITE+TRB, which are more vital to any transportation planner
• I really don’t want to be a member of APA.  It has done little for me, and Planning magazine has published questionable articles on more than one occasion.  One article even made me think that the author was high on drugs.  However, you cannot split APA and AICP.
• While the AICP Commission tried to make AICP actually mean something by requiring CM, their one step forward took two steps back when they attempted to use it as a money-making opportunity for APA and requires approval for all hours.  This means that if you go to Ohio Model User Group meetings or attend Travel Model Improvement Program webinars, you’re not getting any certification maintenance credit for it.  If you are a Professional Engineer or a Professional Transportation Planner (the certification through TCSB), you get credit for these things.

In keeping up with things going on in the field, I received an email from the Florida DOT Weekly Briefs.  In it, they had a link to “State of Transportation Planning 2010″.  They had an overview of some important topics (with suggested reading), but missed two, including:

• FINANCE. one of the biggest topics currently impacting transportation.
• SAFETY.  There are almost 40,000 people killed on America’s roads and another 300,000 injured.  Distracted driving has received a lot of attention because of the crashes that it causes and has the potential to cause.  Toyota has recalled some 8 million vehicles due to problems with sticking gas pedals, brakes, or steering.

Aside from the obvious blunders above, there are a few more that I think need to be pointed out.

First off, this is not a “state of” anything.  This is a very basic introductory document that points people to a few seemingly randomly selected works in that field.  Looking at the travel behavior section, there are several research works (and having met some of these authors, I know they are great resources).  The one thing that we already have as travel demand modelers that illustrates the “state of” is Transportation Research Board Special Report 288.  It is so vitally important to us that we know it by number (and it is the only one, in my case).  SR288 goes through in sometimes painful detail to talk about the state of travel forecasting.

Second off, policy is centered entirely around climate change.  They evidently haven’t been following Climategate, else they would have dropped the link to “Moving Cooler” (considering that areas near Washington DC received an uncharacteristic 5 feet of snow in some places, you’d think that might tip them off).  Obviously, the jury is still out on whether the globe is getting hotter or colder, but if you talk with real people, many feel that Global Warming isn’t occurring.  Stuff like that happens when you mess with data and hard code models for certain outputs.

So yeah, as Marty Brenaman (play-by-play radio host for the Cincinnati Reds) says quite often, “A swing and a miss”.

Here’s the link: TPD_State_of_Transportation.

I have always shied away from anything Tech Crunch, since it has been one of the most likely sites to report hoaxes and rumors.  The above link was posted to PlaNetizen and went out to thousands of readers.  I’m sure there are hundreds saying “Get your head out of your ass!  We have that here!”

Those hundreds are in Tampa.  They have an elevated freeway above the Lee Roy Selmon Cross Town Expressway.  They were finished in 2007 or so.  I’ve heard that they are really, really cool to use.  It was also really, really expensive, and it suffered a setback after one of the support piers sunk, bringing a 3 lane section down to the ground level freeway below.

Another thing that should be pointed out is that in this country there are dozens of bridges that are double-decker.  We have one here in Cincinnati, the Brent Spence Bridge, which carries I-71 and I-75 over the Ohio River.

That bridge needs to be rebuilt.  It is roughly a half of a mile (including the approaches), and the costs are measured in the billions.

Dear author, if you are going to tell people to start a company, make sure it hasn’t already been started first.

EDITS:

(1) I figured I needed to come back to this and bring up an important issue that I only touched on.  That is the potential for problems caused by natural disasters.  After one of the “Big Ones” in California, it was brought up that Cincinnati is somewhat earthquake-prone.  The thing that made the nightly news?  ”Is the Brent Spence Bridge Going to Collapse if Cincinnati gets hit with a Big One?”  Certainly the Tampa Crosstown Expressway sink was a scary, serious issue.  It was one that was not planned for, and required a complete redesign of an area of the elevated expressway.  The reason roads tend to be on the ground is because there is a limited set of catastrophic natural disasters that can happen.

(2) I dogged TechCrunch above, and I figured I should mention that despite my general poor opinion, I did read another article that sourced a number of topics to TechCrunch, and the source data sounds spot-on.  I’m not a TechCrunch reader, and my opinions are still mixed.

I agree with the thoughts of increased tolling and more fees other than the gas tax.  I also agree with $1B per year for technology, but it has to be managed right.

I’m also glad that the performance measures are measurable:

• Congestion (we can measure that – it is the percent of a region’s network that is operating with a demand greater than its capacity)
• Costs (we can measure that, although we have to watch how we do it, as we don’t want to have a system be considered bad if gas prices hit $4/gallon)
• Safety (we DO measure this – it is the number of injuries and deaths on the road)

Traffic Counter on side of road

First off: how these things work

Those that have been around for 30 or more years may remember when some gas stations had a hose that rang a bell to call a station attendant to pump your fuel. Those that don’t should watch Back to the Future. This is the same basic concept for most traffic counters. There are hoses that go across the road, and based on what the sensors feel and the time between them, these little green (or sometimes gray) boxes calculate the number of axles, distance between them (which can be used to derive the type of vehicle), and the speed.

I know that speed is a big issue with a lot of people. After all, some of these counters are being used for speed studies to see if they want to put a cop on a road at a certain time. This does happen, despite my wishes that cops and others would use less-detectable methods for enforcement. There are two other ways that counts, with speed, can be taken. One is by RADAR (the same thing they use for active speed enforcement). Mind you, for speed sampling, RADAR is pretty useful when installed correctly, and the boxes can be hidden quite well. The other is using magnetic loops. There are portable models of these that sit in the lane and are difficult to see (and also susceptible to theft). There are also permanent models that can be completely hidden from view.

One thing I can say with ALL hose counters: WE CANNOT USE THEM FOR SPEED ENFORCEMENT! The units do not have any cameras (etc), so if you speed while going over them, we know you did, but we don’t know who you are!

Second off: How We Use The Data We Get From These Things

This one differs by jurisdiction, but most use it for traffic studies. Speed, count, and vehicle type are very useful for roadway improvement design. Another use is for travel model validation. We (specifically me, since it is my job) use this to ensure that the region’s travel model is accurate so when we use it to plan billions of dollars in improvements, we know we’re not just guessing, which would be a waste of money.

Law enforcement will use the number of speeders per unit of time to plan when to run patrols. As I indicated, I wish they wouldn’t use hose counters for this, but they do, and the data they get is accurate. However, hoses are pretty conspicuous, which is why I wish they wouldn’t use them.

We cannot use the data in court. You cannot be detected to be going 45 MPH in a 25 MPH zone based on a traffic counter. The counters do not have cameras in them, and none that I know of can connect to a camera. A camera would be required to prove who was speeding. Without the connection, it would be difficult to prove, since the times would have to be the same, the counter has to be operating perfectly, and the hoses have to be measured very precisely. Some states also forbid the use of cameras for passive law enforcement (a cop can actively use a RADAR+camera, but not mount one on a pole and get every car that is speeding).

The War Stories

I have two, both given to me by a salesperson for Jamar Tech, one of the leading traffic counter manufacturers.

City of Boston Thinks a Counter is a Bomb. This one is proof that some cops don’t use hose counters, else they would have known what this unit is.

Counter burned, likely by an accelerant. PDF from Jamar, which the salesperson sent me just after I bought 8 counters from him.

Don’t Mess With Them!

It amazes me that 1 month into the season, I’ve had to replace several hoses because of cut or stolen hoses. This is your tax dollars at work. The more hoses we have to replace, the less money we have to improve the roads.

Binomial Logit Mode Choice: A process that compares the utility of two travel mode choices and determines the probability of using each mode.

BPR Equation: An equation used to determine the speed on a network link based on the ratio of the volume to the capacity.

Centroid: A point that is linked to a TAZ that loads traffic onto the network

Centroid Connector: The link that connects a centroid to the rest of the network.

Equilibrium Assignment: An assignment method that iteratively assigns and reassigns traffic based on revised travel times on each link.

Friction Factors: Factors that are used in distribution to refine the distance measure between zones.

The Four Step Process: A process of modeling that began in the 1950s that performs four major steps: trip generation, trip distribution, mode choice, and assignment.

Fratar Model: An iterative gravity model used to determine trip interchanges where total productions and attractions are known, but not necessarily balanced.

Gravity Model Distribution: A process to determine the likelihood of traveling from one zone to another based on the attractiveness of the destination compared to the distance from the origin to the destination.

Link: A piece of roadway in the network

Mode Choice: A process that determines the economic utility of all available forms of travel from one zone to another and splits trips (from the trip distribution step) into a variety of modes (depending on those that exist).

Multinomial Logit Mode Choice: A process that compares the utility of multiple travel mode choices where each is a separate and distinct choice.

Nested Logit Mode Choice: A process that compares the utility of multiple travel mode choices as a string of choices and sub-choices.  An example would be a structure illustrated below (click on the image for a larger view).

Network: A collection of links, nodes, centroids, etc. that simulate the major travel facilities in the model area.

Traffic Analysis Zone (AKA: zone, TAZ): A geographic division, much like a census block, that defines an area of homes and businesses used in the travel demand model.

Trip Generation: A process of counting the homes, employees (usually by type), schools, and other activities that generate traffic to and from them.  Trips are generated as attractions and productions.

Trip Distribution: A process of determine where generated trip productions are linked to generated trip attractions.  This determines the number of people that will go from one zone to another.

• Malfunction Junction (I-275 and I-4, Tampa, FL)
• The Riddle in the Middle (Alaska Way, Seattle, WA)
• Spaghetti Junction (I-85 and I-285, Atlanta, GA)

I’ve also started calling a strech of Columbia Parkway (Cincinnati, OH) “The Suicide Side”, which is a 45 MPH arterial that everyone goes 60 MPH.  The divider is a double-yellow line… only.

Got any more?  Add ‘em in the comments.

The socioeconomic data for most models includes the following attributes for every zone in the model area:

• Population
• Households or Dwelling Units
• Dwelling Unit Occupancy
• Auto Ownership Information
• Hotel Information
• School Enrollment
• Employment Information (number, type)
• Special Cases (airports, amusement parks, etc)

Many models have several fields that describe these basic attributes.  In some, auto occupancy may be defined as a percentage of households with 0 autos, a percentage with 1 auto, and a percentage with 2 or more autos.  In others, there may be percentages based on the number of vehicles compared to the number of workers in a household.

The simple part of trip generation models is that once the socioeconomic data is in place and trip production factors are in place, trip generation is simply applying factors to determine the number of trips generated by and attracted to each use.  These factors are determined by a trip survey.

Trip generation is normally done with a cross-classification model.  This is where multiple factors are used to determine the rate of trips per unit (household, employee, etc).

This spreadsheet is an example of trip generation calculations. It is a cross-classification model based on household size and auto ownership.  For each purpose, the dwelling units with each criteria of auto ownership is is calculated and then applied to the trip rate in the ‘Production Table’ tab.  For attractions, I simply applied a rate per employee to the employees by sector.  The SEData sheet indicates the inputs and outputs.

The final output is a set of productions by trip purpose and a set of attractions.  These are intentionally close with attractions higher than productions.  The reason why will be discussed in the Trip Distribution post.

A trip purpose is basically a reason for travel.  Most models include home-based work (which are any trip going to or coming from the home and work), home-based school, home-based other (trips for purposes other than school and work), and non-home-based (trips that do not go to or from the home).  Some models may include home-based shopping or home-based social-recreation separately, and some combine it into home-based other.

trip-generation-worksheet

1. Trip Generation
2. Trip Distribution
3. Mode Choice
4. Trip Assignment

The first step in the process, Trip Generation, uses socioeconomic data to determine the number of trips produced by a traffic analysis zone (or census tract, census blockgroup, or other geographic division).  The socioeconomic data normally includes population, auto ownership, and employment information at the very least.

The second step is trip distribution.  Once the number of trips are known, trip distribution determines where the trips will go.  This normally uses a gravity model, which will be discussed in a future post.  This step takes in several factors, including the number of trip productions, the number of trip attractions, and an impedance value.  The impedance value is the resistance to travel, which could include distance, time, tolls, or a combination of those.  Each model is different in this regard.

The third step is mode choice.  This step determines what vehicle trips will utilize when going from one zone to another.  This step can be particularly complex or extremely simple, depending on the area included in the model.  Some models have very simple mode choice steps because transit isn’t available, or it doesn’t have a significant effect.  Other models can be extremely complex, such as a model for Chicago or New York, which would include autos, bus transit, subway transit, urban rail (such as Chicago’s Metra Rail), and intercity rail transit (such as the Amtrak service that connects New York to Boston, Baltimore, and Washington, DC).

The final step is trip assignment.  This step takes all of the trips from mode choice (which it now knows if they are trips that will drive alone, share a ride, use the bus, or use another mode of transportation) and assigns them to a transportation network.  Prior to the mid-1980s, these networks were largely text based, but with the advent of Geographic Information Systems and personal computers in the mid-1980s and moving forward, these networks became graphical.

These four steps represent the basic building blocks of most transportation models.  These steps are a basic way to ask:

• How many people are going to travel?
• Where are they going to go?
• What transportation mode are they going to use to get there?
• What route will they take to get there?