It’s offical now, Augmented-Intelligence was awarded one of the ACT Government’s Innovation Grants (ICON) as part of the most recent round. The Grant will be used to further our product trials and build out the FatigueM8 portal.
The ICON Grant, along with matched contributions from Augmented-Intelligence and our partners will see close to $100k invested into our project “Fatigue and cardiovascular monitoring for heavy vehicle drivers using an ECG monitor embedded in the steering wheel.”.
The ICON Grants rounds are always very competitive and we are ever so thankful to the the ACT Government and CBR Innovation for support of our journey. We’ll be putting the collective funds to good use, purchasing the hardware to create an additional five (5) FatigueM8 units. The additional units will allow our current trial plans to be doubled from five (5) to ten (10) vehicles, with potential for up to 20 drivers to participate.
As well as the additional units, we’ll also be creating a FatigueM8 portal, where our drivers will be able to login and review their data in an easy to use manner. Watch this space for further developments.
This week we welcome into the FatigueM8 team, Katie Speer, who’s a Ph. D Candidate at the University of Canberra (UC). Katie is studying within the Department of Sport & Exercise Science, Faculty of Health and has published several research papers investigating Heart Rate Variability (HRV) in connection with the autonomic nervous system, maternal exercise influence on childhood HRV and also investigated the ability of commercially available devices to measure HRV in children. All of her papers are available via the link below.
Katie joins the FatigueM8 team for a five (5) month internship to undertake a research project titled “The Validation of the FatigueM8 System for using heart rate variability to detect truck driver fatigue”. Katie will be supported by UC Associate Professor Andrew Mc Kune. The FatigueM8 team was introduced to Professor Mc Kune as we researched the potential cause of HRV observations we’d captured after a trip to Threadbo mountain biking.
Professor Mc Kune was a review on the paper we referenced here (doi: 10.3389/fphys.2017.00301) and noting he was a local Canberrian we reached out.
Professor Mc Kune’s “interests include investigating bioenergetic and stress responses (inflammation, autonomic nervous system, HPA-axis) in chronic disease. How do exercise and/or nutrition interventions impact these systems to prevent and treat chronic disease? Methodologies include heart rate variability, salivary bioscience, cardiopulmonary exercise testing, telomere length and ultrasound.”
We’re super excited to have Katie and Andrew onboard; as part of the project they’ll be undertaking we’re going to be expanding our in-cab trials and data collection.
Watch this space for some exciting announcements over the coming weeks.
A couple of Friday’s have past since the last post, but it’s not because of a lack of activity on our side. We been busily working away to grow out our capability. One of the initial features that we had in mind from the outset of FatigueM8, was to add context to the ECG observations by overlaying additional information sets. This week we’ve brought online an additional data stream, leveraging AWS Rekognition service.
As we have explored in previous posts, we’ve had our fair share of challenges with our forward facing camera. Adding in the Rekognition classifications threw up a couple more curve balls to us. The pictures below are from two (2) of our trucks were the FatigueM8 unit had shifted since being installed. Being prototypes we’d used “suckers” to connect the unit to the wind screen and with the changing of temperatures the suction was lost resulting the cameras pointing every which way.
A quick trip to the local hardware store has us back up and pointing in the right direction! A little 25 cms elastic strap and a couple of cable ties have the install back on track.
The resulting photos (below) are back to being captured correctly and ready for processing through rekognitition. Now when an image is uploaded from the truck into the cloud it’s classified by rekognition and then depending on what’s been found, boxes are drawn around the objects.
If a person is detected in the image, we’ll blur them out or don’t use the image just to be safe.
As we run all the images through this process we’ll be able to build up a picture (no pun intended) about the traffic, road and environmental conditions our drivers are driving in; and this will add more context to the ECG observations we collecting.
To kick off this week’s edition we start with a nice time series set of images captured from one of our units based in Canberra. On Thursday morning Canberra awoke to a thunderstorm and in this set of images (which span 7 minutes) you can see the rain falling and then freezing on the bonnet of the truck.
This week saw the commissioning of a new FatigueM8 unit, into one of Multiquip’s Kenworth T408 SAR’s, which is setup pulling a B-Double between Marulan and several locations in Sydney.
It’s the first installation into a T408, which is an older model Kenworth (circa 2009/10), fortunately we’d been working with steering wheel in our lab which was the same model as is installed in the T408.
The installation was straight forward and with the steering wheel cover installed powered it up for testing. We observed the ECG monitor “cycling” through the start up sequence over and over again. When measured the electrical current coming through the steering column we noticed some variance in the electrical current. The voltage appears to be bouncing between 11.5 volts and up to 14.5 volts; our ECG monitor operates up to 13.2 volts before switching off to protect the electrical circuit.
In the spirit of rapid prototyping, we setup a voltage regulator in the lab, with the aim of providing a stable 12 volts to the ECG unit. We assembled a linear voltage regulator (below) which was able to drop the power by at least ~1 volt.
We attached the regulator to the T408’s horn cover (pictured below) and set about testing our theory.
We had some success with the voltage regulator in place, the ECG unit powered up and connected to the FatigueM8 on-board unit. We were able to capture the ECG signal for a period of time, before it shutdown and disconnected. We tested all the connections and power supplies and all appeared/reported working at the expected voltage. Our investigations are continuing, watch this space for the next update!
On the positive side, we’re starting to collect data from the front camera and GPS which will be fed into our models.
On Wednesday this week I had the opportunity to present our Augmented-Intelligence story to the AWS Canberra User group (Amazon Web Services). It was great chance to reflect on our journey thus far, revisiting the promises and commitments we made to the Australian Trucking community at the 2018 FatigueHACK.
From the beginning, leveraging the cloud (and AWS) were front of mind. As I flicked through the official photos from the event I came across this one (below) from about midway through the first day. AWS Greengrass and AWS Sagemaker are called out as part of the solution; as it happens both of these products are still in the roadmap, but not yet implemented.
But that’s enough of the past, let’s look at what we have currently implemented. The picture below appeared in our presentation and I’ve marked up what we have currently implemented and being uploaded into AWS in the cloud.
We have three (3) FatigueM8 units installed and commissioned, and in each we are capturing the drivers ECG observations, the view out the front wind screen and geocoding each of the observations. From these we’re able to derive vehicle speed, distance travelled, driving time, road conditions, traffic conditions, drivers Heart Rate and a whole host of other statistics.
In my presentation from Wednesday night drilled into all of these areas, highlighting the how we leverage the various AWS Services (AWS S3, AWS Lambda, AWS Anthea and AWS QuickSight) to capture, store, analyse and present the data that we feed into our models to calculate the FatigueM8 Fatigue Score (aka FFS).
Wednesday’s presentation was recorded and is now available to be watched (or re-watched) via the following link. The FatigueM8 overview starts about 17 mins into the recording. You can check out the presentation here https://www.twitch.tv/videos/659965587
Looking back, we promised to spend the prize money on research (and not beer) which we’ve done; oh boy did we stretch that $6k a loooonnnnngggg way. We committed to continue to repay generosity shown to us during the competition, provide objective data (eg ECG recordings) to contribute to a better understanding of how fatigue effects individuals and to help everyone “keep on trucking” in a healthy and sustainable manner.
As for the sky being the limit, well being born in the cloud does have it’s advantages, but its important that we remain focused on supporting those who’s hands are on the steering wheel and feet firmly on the pedals!
Our FatigueM8 dataset is continually expanding with several vehicles based in Canberra. Each FatigueM8 unit is fitted with a GPS and each of the artefacts we capture (ECG, Steering wheel movement and front facing camera images) or create (ECG Analysis, vehicle Telemetric’s) is geotagged to add extra depth to our analysis. The images displayed (below) plot the on the map the position of the vehicle when the driver had 2-hands on the steering wheel and FatigueM8 was able to capture a valid ECG signal.
An example of a valid ECG segment captured whilst driving is shown below, along with the heart beat wave form commonly referred too as the QRS Complex.
In the example (above), the R-peak is clearly visible throughout the 10 second sample. The valid sample was collected here (see below) and whilst the car was travelling at 86 kms per hour (in a 90 kms p/hr speed zone). The view out the front of the vehicle was captured, with the traffic conditions being quite light and getting closer to sunset.
Below is an example of an invalid ECG signal, captured just 12 seconds later and as can be clearly seen, there is a mix of R-Peaks, as well as high and low signals. In this 10 second snippet there is not enough R-Peaks identifiable to make any calculations or useful observations in relation to heart rate variation.
The sharp variances up and down correspond to the driver having 2 hands, then 1 hand on the wheel. When we add some more context to the ECG signal from the GPS and front facing camera image, we get more an understanding of what may have happened. One possible scenario would be based on the front facing camera showing the vehicle has changed lanes, which if done correctly would have meant the driver took a hand off the steering wheel to put an indicator on and then switch it off again.
Now back to the pattern. By plotting the location of the valid ECG snippet collections onto the a map (as per the image below), we saw that a high proportion of the samples have been collected on Canberra’s main arterial roads.
We have a couple of hypothesis around why this is the case and we’ll explore them more over the coming weeks.
It’s no secret that fatigue is one to the top three (3) killers on our roads in general. However when it comes to fatal heavy vehicle accidents in 2019, fatigue was the largest cause (at 35%) reported in the NTi Major Accident Report 2020.
Crunching the numbers, sadly 50 drivers lost their lives during 2019 and fatigue was responsible for 18 (or 35%) of the 50 lives lost.
The NTi also found that 9% of fatal accidents were caused by a Medical Events, and these events were almost always caused by drivers suffering heart attacks whilst driving. That means that in 2019, 5 drivers lost their lives to cardiovascular related diseases.
These are quite sobering numbers indeed. Fortunately, the trucking industry as a whole has long recognised the need to reduce to 0 the lives lost on our roads; and through the sadness of the lives already lost, there is hope and positivity that together we’ll be able to make sure every drivers gets home safely.
I’ve included a link to the NTi’s report if you’d like to drill into the details and other non-fatigue related findings.
In the last instalment of FatigueM8 Friday’s we looked at the impact of a subtle change to the FatigueM8 units positioning. The image below on the left as you’re looking at it show’s the images being captured by the Front, Sky Facing Camera as a result of the slight changes in unit angle. The Picture on the right shows the actual camera angel.
Problem solved, after a quick trip back to the drawing board, our design team made what looks like a simple tweak to straighten up the camera angel. The resulting image being captured is back to perfecto (as can be seen below).
While the differences between the Before and After camera unit look relatively minor, we’ve actually redesigned the while camera housing. As can be seen in the series of imaged below, the new camera housing allows for the camera angel to be adjusted quite significantly; allowing greater flexibility in our deployment options.
Part of our FatigueM8 solution is a forward facing camera, that captures the road and traffic conditions our drivers are operating in. The camera position has been refined through a few iterations. The current positioning allows for the installation of the FatigueM8 unit on an angle, catering for the shape of dashboards that generally slope down towards the front of the vehicle. The resulting photographs capture the road and traffic quite well.
In early May 2020 we noticed a change in the photo’s that were being captured, there was a lot more sky and a lot less road (as can be seen below). Chatting with our driver, they’d adjusted the angle of the unit to decrease the reflection of the blue light emitted by the LCD screen and resulted in a beautiful selection of “sky shots”.
As discussed in last week’s post, we have removed the LCD screen in the latest iteration to resolve the light emission. Now that we’ve installed the new version we’ve noticed the “sky shots” returning. The cause is again because of the camera angle. In our original design we catered for the “dashboard angle”, however with the new design, the length of the unit is shorter, which allows for the unit to sit essentially flat, pointing the camera skywards.
The change in the camera angle can be seen clear in the images below, using our “view finder jig”.
Our solution is currently will be to create an adjustable camera mount, which will adapt to the different FatigueM8 mounting positions. It’s just a paper sketch at the moment, but within the week it’s be transformed into a 3D print and fitted into the truck for testing.
My Friday afternoon this week concluded with a visit to the Elvin yard in Mitchell; all the trucks had been washed and neatly parked up for the night. My visit was two-fold this, firstly to check on the newly installed FatigueM8 unit, which we’d deployed earlier in the week and also to ‘tweak’ the steering wheel cover installed some 8 weeks earlier.
Job # 1: The new FatigueM8 model has some subtle, yet major changes to the unit. The most notable is the removal of the LCD panel on the front of the unit. During development, the LCD has been an invaluable part of the FatigueM8 compute model; displaying messages, errors and providing feedback, allowing us to see that inside the black box, all is as it should be. Now that the FatigueM8 units are being deployed and used in production, the need for the LCD has reduced. From the outset our mantra has always been “we want the driver to just drive and we’ll take care of the rest”, with the transition from Summer into Autumn and the end of daylight savings meant they were driving more often in dark conditions. While chatting to one of our drivers he mentioned that the blue glow emitted by the LCD in the dark cabin conditions more noticeable. Wanting to ensure we minimise distraction and light pollution in-cab for our drivers, we tweaked the design, reprinted the unit (they are 3D printed locally in Canberra) and installed it.
Job # 2: Since installing our first persistent FatigueM8 unit into the Kenworth T359A some eight (8) weeks ago, we’ve been checking in with the regular with the driver, to get feedback on the unit’s performance. Overall the use of the unit has been seamless, with the only issue being the stitching on one section of the cover had loosened (as can be seen in the picture on the top left below).
After an hour of careful stitching, we fixed the troublesome section and made sure the rest of the wheel won’t have the same issue. When I shared the photo on the bottom-right with the driver he commented “It looks like a racing car steering wheel!” which I think is a good thing 🙂