ScrAPD

https://badge.fury.io/py/scrapd.svg https://circleci.com/gh/scrapd/scrapd.svg?style=svg https://coveralls.io/repos/github/scrapd/scrapd/badge.svg?branch=master

Extract data from APD news site.

ScrAPD is a small utility designed to help organizations retrieving traffic fatality data in a friendly manner.

Installation

ScrAPD requires Python 3.7+ to work.

pip install scrapd

Quickstart

Collect all the data as CSV:

scrapd --format csv

By default, scrapd does not display anything until it is done collecting the data. If you want to get some feedback about the process, you can enable logging, by adding the -v to the command you want to use. Multiple -v options increase the verbosity. The maximum is 3 (-vvv):

scrapd -v --format csv

To save the results to a file, use the shell redirection:

scrapd -v --format csv > results.csv

Note

The logs are displayed to stderr and will not appear in the result file generated by the redirection. If you want to include this information add &2>1.

Examples

Retrieve the traffic fatalities that happened between January 15th 2019 and January 18th 2019, and output the results in json:

scrapd --from "Jan 15 2019" --to "Jan 18 2019" --format json

[
  {
    "case": "19-0161105",
    "crash": 2,
    "date": "2019-01-16",
    "fatalities": [
      {
        "age": 58,
        "dob": "1960-02-15",
        "ethnicity": "White",
        "first": "Ann",
        "gender": "Female",
        "generation": "",
        "last": "Bottenfield-Seago",
        "middle": ""
      }
    ],
    "latitude": 0.0,
    "link": "http://austintexas.gov/news/traffic-fatality-2-3",
    "location": "West William Cannon Drive and Ridge Oak Road",
    "longitude": 0.0,
    "notes": "The preliminary investigation shows that the grey, 2003 Volkwagen Jetta being driven by Ann Bottenfield-Seago failed to yield at a stop sign while attempting to turn westbound on to West William Cannon Drive from Ridge Oak Road. The Jetta collided with a black, 2017 Chevrolet truck that was eastbound in the inside lane of West William Cannon Drive. Bottenfield-Seago was pronounced deceased at the scene. The passenger in the Jetta and the driver of the truck were both transported to a local hospital with non-life threatening injuries. No charges are expected to be filed.",
    "time": "15:42:00"
  },
  {
    "case": "19-0150158",
    "crash": 1,
    "date": "2019-01-15",
    "fatalities": [
      {
        "age": 31,
        "dob": "1987-07-09",
        "ethnicity": "White",
        "first": "David",
        "gender": "Male",
        "generation": "",
        "last": "Sell",
        "middle": "Hilburn"
      }
    ],
    "latitude": 0.0,
    "link": "http://austintexas.gov/news/traffic-fatality-1-4",
    "location": "10500 block of N IH 35 SB",
    "longitude": 0.0,
    "notes": "The preliminary investigation shows that a 2000 Peterbilt semi truck was travelling southbound in the center lane on IH 35 when it struck pedestrian David Sell. The driver stopped as soon as it was possible to do so and remained on scene. He reported not seeing the pedestrian prior to impact given that it was still dark at the time of the crash. Sell was pronounced deceased at the scene at 6:24 a.m. No charges are expected to be filed.",
    "time": "06:20:00"
  }
]

Do the same research but output as CSV:

scrapd --from "Jan 15 2019" --to "Jan 18 2019" --format csv


crash,case,date,time,location,first,middle,last,generation,ethnicity,gender,dob,age,link,notes
2,19-0161105,2019-01-16,15:42:00,West William Cannon Drive and Ridge Oak Road,Ann,,Bottenfield-Seago,,White,Female,1960-02-15,58,http://austintexas.gov/news/traffic-fatality-2-3,"The preliminary investigation shows that the grey, 2003 Volkwagen Jetta being driven by Ann Bottenfield-Seago failed to yield at a stop sign while attempting to turn westbound on to West William Cannon Drive from Ridge Oak Road. The Jetta collided with a black, 2017 Chevrolet truck that was eastbound in the inside lane of West William Cannon Drive. Bottenfield-Seago was pronounced deceased at the scene. The passenger in the Jetta and the driver of the truck were both transported to a local hospital with non-life threatening injuries. No charges are expected to be filed."
1,19-0150158,2019-01-15,06:20:00,10500 block of N IH 35 SB,David,Hilburn,Sell,,White,Male,1987-07-09,31,http://austintexas.gov/news/traffic-fatality-1-4,The preliminary investigation shows that a 2000 Peterbilt semi truck was travelling southbound in the center lane on IH 35 when it struck pedestrian David Sell. The driver stopped as soon as it was possible to do so and remained on scene. He reported not seeing the pedestrian prior to impact given that it was still dark at the time of the crash. Sell was pronounced deceased at the scene at 6:24 a.m. No charges are expected to be filed.

Retrieve all the traffic fatalities from 2019 (as of Jan 20th 2019) in json, and enabling the logging to follow the progress of the process:

scrapd -v --from "1 1 2019" --to "Jan 20 2019" --format json

Fetching page 1...
Fetching page 2...
Fetching page 3...
Fetching page 4...
Fetching page 5...
Fetching page 6...
Fetching page 7...
Fetching page 8...
Fetching page 9...
Fetching page 10...
Fetching page 11...
Total: 3
[
  {
    "case": "19-0081623",
    "crash": 3,
    "date": "2019-01-08",
    "fatalities": [
      {
        "age": 15,
        "dob": "2003-02-18",
        "ethnicity": "Hispanic",
        "first": "Jesus",
        "gender": "Male",
        "generation": "",
        "last": "Servantez",
        "middle": ""
      }
    ],
    "latitude": 0.0,
    "link": "http://austintexas.gov/news/traffic-fatality-3-4",
    "location": "3600 block of South Capital of Texas Highway SB",
    "longitude": 0.0,
    "notes": "The preliminary investigation shows that the driver of a silver, 2018 KIA was traveling in the center lane of the 3600 block of South Capital of Texas Highway SB when the car collided with Jesus Servantez, a pedestrian in the roadway. The driver remained at the scene and told investigators that he did not see Servantez prior to the crash.\n\n\tJesus Servantez was transported to Saint David\u2019s South Austin Medical Center where he succumbed to his injuries on January 21, 2019. No charges are expected to be filed.",
    "time": "21:37:00"
  },
  {
    "case": "19-0161105",
    "crash": 2,
    "date": "2019-01-16",
    "fatalities": [
      {
        "age": 58,
        "dob": "1960-02-15",
        "ethnicity": "White",
        "first": "Ann",
        "gender": "Female",
        "generation": "",
        "last": "Bottenfield-Seago",
        "middle": ""
      }
    ],
    "latitude": 0.0,
    "link": "http://austintexas.gov/news/traffic-fatality-2-3",
    "location": "West William Cannon Drive and Ridge Oak Road",
    "longitude": 0.0,
    "notes": "The preliminary investigation shows that the grey, 2003 Volkwagen Jetta being driven by Ann Bottenfield-Seago failed to yield at a stop sign while attempting to turn westbound on to West William Cannon Drive from Ridge Oak Road. The Jetta collided with a black, 2017 Chevrolet truck that was eastbound in the inside lane of West William Cannon Drive. Bottenfield-Seago was pronounced deceased at the scene. The passenger in the Jetta and the driver of the truck were both transported to a local hospital with non-life threatening injuries. No charges are expected to be filed.",
    "time": "15:42:00"
  },
  {
    "case": "19-0150158",
    "crash": 1,
    "date": "2019-01-15",
    "fatalities": [
      {
        "age": 31,
        "dob": "1987-07-09",
        "ethnicity": "White",
        "first": "David",
        "gender": "Male",
        "generation": "",
        "last": "Sell",
        "middle": "Hilburn"
      }
    ],
    "latitude": 0.0,
    "link": "http://austintexas.gov/news/traffic-fatality-1-4",
    "location": "10500 block of N IH 35 SB",
    "longitude": 0.0,
    "notes": "The preliminary investigation shows that a 2000 Peterbilt semi truck was travelling southbound in the center lane on IH 35 when it struck pedestrian David Sell. The driver stopped as soon as it was possible to do so and remained on scene. He reported not seeing the pedestrian prior to impact given that it was still dark at the time of the crash. Sell was pronounced deceased at the scene at 6:24 a.m. No charges are expected to be filed.",
    "time": "06:20:00"
  }
]

The “Case” field is assigned by the Austin Police Department, while the “ID” field is a key assigned by ScrAPD to label a separate record for each deceased person.

Speed and accuracy

ScrAPD executes all the requests in an asynchronous manner. As a result it goes very fast.

It parses the information using both the text of the report itself and the Twitter tweet stored in the page metadata. Combining these two methods provides a high degree of confidence in the parsing and allows us to reach more than 95% of success rate.

Some statistics:

  • 99% of entries detected

  • 95% of entries correctly parsed

    • 5% of entries failed the parsing only partially

    • 0 complete failure

  • processing time: ~2m00s for ~140 entries

Who uses ScrAPD?

The Austin Pedestrian Advisory Council used ScrAPD to compile a detailed presentation of the status of the traffic deaths in Austin, TX:

The following organizations also use ScrAPD:

See more at http://docs.scrapd.org/who-is-using-us/