Visualizing Uncertain Tropical Cyclone Predictions using Representative Samples from Ensembles of Forecast Tracks
Le Liu, Lace M. K. Padilla, Sarah Creem-Regehr, Donald House
Updated: June 28th, 2018
- Click here to view an online version of this analysis.
- Download the data files and code to generate the analysis here.
- Link to the Bitbucket repository that contains the research codes and pseudocodes of our technique.
Participants
7 tracks: 50
15 tracks: 51
63 tracks: 50
Full Vis: 50
Males: 49
Female: 152
Age: Mean = 23.53, SD = 6.6
Results: H1
VisNumD0 = 7 tracks (referent)
VisNumD1 = 15 tracks
VisNumD2 = 63 tracks
| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 5.4875833 | 0.1338470 | 40.9989179 | 0.0000000 |
| Distance | -2.1011775 | 0.0506198 | -41.5089741 | 0.0000000 |
| Time48 | -0.1769049 | 0.0782433 | -2.2609605 | 0.0237617 |
| Vis15 | 0.1199898 | 0.1883581 | 0.6370304 | 0.5241051 |
| Vis63 | 0.5728231 | 0.1892883 | 3.0261942 | 0.0024765 |
| Distance:Time48 | 0.0884836 | 0.0715873 | 1.2360248 | 0.2164493 |
| Distance:Vis15 | -0.0314148 | 0.0712355 | -0.4409991 | 0.6592136 |
| Distance:Vis63 | -0.3519816 | 0.0715873 | -4.9168193 | 0.0000009 |
| Time48:Vis15 | -0.0337828 | 0.1101089 | -0.3068125 | 0.7589861 |
| Time48:Vis63 | -0.6004842 | 0.1106527 | -5.4267473 | 0.0000001 |
| Distance:Time48:Vis15 | 0.0323620 | 0.1007422 | 0.3212354 | 0.7480320 |
| Distance:Time48:Vis63 | 0.5885698 | 0.1012397 | 5.8136279 | 0.0000000 |
Confidence intervals using Wald method
## 2.5 % 97.5 %
## .sig01 NA NA
## .sigma NA NA
## (Intercept) 5.22524793 5.74991864
## Distance -2.20039054 -2.00196443
## Time48 -0.33025893 -0.02355095
## Vis15 -0.24918528 0.48916495
## Vis63 0.20182488 0.94382130
## Distance:Time48 -0.05182482 0.22879208
## Distance:Vis15 -0.17103375 0.10820419
## Distance:Vis63 -0.49229007 -0.21167317
## Time48:Vis15 -0.24959236 0.18202676
## Time48:Vis63 -0.81735950 -0.38360891
## Distance:Time48:Vis15 -0.16508909 0.22981299
## Distance:Time48:Vis63 0.39014367 0.78699590Effect size
## R2m R2c
## 0.3777288 0.5606085The results of this analysis revealed a significant three-way interaction between distance, time point, and the 7 vs. 63 track display, b = 0.589, t = 5.814, p = 0.000, 95 % CI [0.39, 0.78] (highlighted in orange). To break down the three-way interaction, the same equation as above was computed but on each of the visualization types separately.
Post-Hoc: 63-track
| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 6.0604064 | 0.1161452 | 52.179558 | 0 |
| Distance | -2.4531591 | 0.0521412 | -47.048379 | 0 |
| Time48 | -0.7773891 | 0.0805949 | -9.645642 | 0 |
| Distance:Time48 | 0.6770534 | 0.0737388 | 9.181780 | 0 |
Confidence intervals using Wald method
## 2.5 % 97.5 %
## .sig01 NA NA
## .sigma NA NA
## (Intercept) 5.8327659 6.2880468
## Distance -2.5553540 -2.3509642
## Time48 -0.9353522 -0.6194261
## Distance:Time48 0.5325280 0.8215788Effect size
## R2m R2c
## 0.3999559 0.5277020There is a significant between interaction time point and distance (highlighted in orange). To break down the interaction, the same equation as above was computed but on each time point separately.
Post-Hoc: 63 track, 24-hr
| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 6.060406 | 0.1192183 | 50.83454 | 0 |
| Distance | -2.453159 | 0.0555943 | -44.12608 | 0 |
Confidence intervals using Wald method
## 2.5 % 97.5 %
## .sig01 NA NA
## .sigma NA NA
## (Intercept) 5.826743 6.294070
## Distance -2.562122 -2.344196Effect size
## R2m R2c
## 0.4390660 0.5492331The 63 track display at 24 hours has an overall slope of -2.45, meaning that for every one unit change in distance, damage ratings decrease by 2.45 on average on the Likert scale from 1-7, conditional R-squared = .55.
Post-Hoc: 63 track, 48-hr
| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 5.283017 | 0.1181877 | 44.70025 | 0 |
| Distance | -1.776106 | 0.0474583 | -37.42452 | 0 |
Confidence intervals using Wald method
## 2.5 % 97.5 %
## .sig01 NA NA
## .sigma NA NA
## (Intercept) 5.051374 5.514661
## Distance -1.869122 -1.683089Effect size
## R2m R2c
## 0.3400946 0.5145997Whereas, for the 63 track display at 48 hours, the average slope is -1.77. The original time point and distance interaction (highlighted in orange) indicates that there is a significant difference between the 24- and 48-hr slopes of .68 (b = 0.67, t = 9.18, p ¡ 0.000, 95 % CI [0.53, 0.82], conditional R-squared for the model = 0.527).
Post-Hoc: 15 track
| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 5.6075731 | 0.1389902 | 40.345095 | 0.0000000 |
| Distance | -2.1325923 | 0.0524586 | -40.652844 | 0.0000000 |
| Time48 | -0.2106877 | 0.0810855 | -2.598341 | 0.0093675 |
| Distance:Time48 | 0.1208456 | 0.0741877 | 1.628917 | 0.1033307 |
Confidence intervals using Wald method
## 2.5 % 97.5 %
## .sig01 NA NA
## .sigma NA NA
## (Intercept) 5.33515732 5.8799889
## Distance -2.23540928 -2.0297753
## Time48 -0.36961238 -0.0517631
## Distance:Time48 -0.02455964 0.2662508Effect size
## R2m R2c
## 0.3509818 0.5423804There is not a significant between interaction time point and distance (highlighted in orange).
Post-Hoc: 7 track
| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 5.4875833 | 0.1432303 | 38.313002 | 0.0000000 |
| Distance | -2.1011775 | 0.0464388 | -45.246170 | 0.0000000 |
| Time48 | -0.1769049 | 0.0717806 | -2.464523 | 0.0137196 |
| Distance:Time48 | 0.0884836 | 0.0656744 | 1.347308 | 0.1778809 |
Confidence intervals using Wald method
## 2.5 % 97.5 %
## .sig01 NA NA
## .sigma NA NA
## (Intercept) 5.20685703 5.76830954
## Distance -2.19219584 -2.01015912
## Time48 -0.31759236 -0.03621752
## Distance:Time48 -0.04023577 0.21720303Effect size
## R2m R2c
## 0.3810158 0.6124635There is not a significant between interaction time point and distance (highlighted in orange).
In sum, the three-way interaction indicates an interaction between time point and distance for the 63 track display but not the 7 track display, which supports H1.
Results: H2
VisFull: 15 unannotated is the referant| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 5.6075731 | 0.1327453 | 42.2431115 | 0.0000000 |
| Distance | -2.1325923 | 0.0526726 | -40.4877162 | 0.0000000 |
| TimeD | -0.2106877 | 0.0814162 | -2.5877866 | 0.0096595 |
| VisFull | -0.8778597 | 0.1886665 | -4.6529716 | 0.0000033 |
| Distance:TimeD | 0.1208456 | 0.0744903 | 1.6223002 | 0.1047391 |
| Distance:VisFull | 0.6818169 | 0.0748618 | 9.1076744 | 0.0000000 |
| TimeD:VisFull | 0.0386274 | 0.1157142 | 0.3338177 | 0.7385172 |
| Distance:TimeD:VisFull | -0.1044757 | 0.1058706 | -0.9868246 | 0.3237286 |
Confidence intervals using Wald method
## 2.5 % 97.5 %
## .sig01 NA NA
## .sigma NA NA
## (Intercept) 5.34739718 5.86774906
## Distance -2.23582862 -2.02935592
## TimeD -0.37026055 -0.05111493
## VisFull -1.24763923 -0.50808025
## Distance:TimeD -0.02515266 0.26684383
## Distance:VisFull 0.53509043 0.82854327
## TimeD:VisFull -0.18816815 0.26542300
## Distance:TimeD:VisFull -0.31197817 0.10302682Effect size
## R2m R2c
## 0.2965322 0.4866122There is a significant interaction between distance and VisFull (15 annotated vs. 15 unannotated) (highlighted in orange). To break this down, we recenter distance around the farthest time point (1.78).
Post-Hoc: re-centering
| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 1.8137676 | 0.1257721 | 14.421069 | 0.0000000 |
| DistanceCenter1.78 | -2.0721695 | 0.0372925 | -55.565297 | 0.0000000 |
| VisFull | 0.2621046 | 0.1787557 | 1.466273 | 0.1425739 |
| DistanceCenter1.78:VisFull | 0.6295790 | 0.0530026 | 11.878263 | 0.0000000 |
This analysis revealed that at the farthest distance there is not a significant difference between the two Vis types (highlighted in blue). Overall, the 15 track elicited greater damage ratings but this difference diminished at the farthest time point.
Results: Size and Intensity for 15 annotated
| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 4.0142793 | 0.1245727 | 32.224385 | 0.0000000 |
| Distance | -1.7950592 | 0.0372792 | -48.151826 | 0.0000000 |
| Time48 | -0.2318941 | 0.0641540 | -3.614645 | 0.0003008 |
| SizeMean | 0.0135490 | 0.0031318 | 4.326301 | 0.0000152 |
| IntenistyMean | 0.0100161 | 0.0009947 | 10.069201 | 0.0000000 |
| Distance:Time48 | 0.0691249 | 0.0527207 | 1.311154 | 0.1898056 |
Confidence intervals using Wald method
## 2.5 % 97.5 %
## .sig01 NA NA
## .sigma NA NA
## (Intercept) 1.56725888 2.0602763
## DistanceCenter1.78 -2.14526146 -1.9990775
## VisFull -0.08825008 0.6124593
## DistanceCenter1.78:VisFull 0.52569579 0.7334622Effect size
## R2m R2c
## 0.2952415 0.4853052As predicted, both size and intensity account for a significant proportion of variance in damage ratings, such that as both size and intensity increase damage also increases.
15 annotated
15 unannotated
Results: 15 annotated vs. 63
VisFull: 63 is the referant| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 6.0604064 | 0.1218563 | 49.734029 | 0e+00 |
| Distance | -2.4531591 | 0.0527825 | -46.476708 | 0e+00 |
| TimeD | -0.7773891 | 0.0815862 | -9.528441 | 0e+00 |
| VisFull | -1.3306930 | 0.1723309 | -7.721733 | 0e+00 |
| Distance:TimeD | 0.6770534 | 0.0746458 | 9.070215 | 0e+00 |
| Distance:VisFull | 1.0023837 | 0.0746458 | 13.428535 | 0e+00 |
| TimeD:VisFull | 0.6053288 | 0.1153803 | 5.246380 | 2e-07 |
| Distance:TimeD:VisFull | -0.6606835 | 0.1055651 | -6.258541 | 0e+00 |
## 2.5 % 97.5 %
## .sig01 NA NA
## .sigma NA NA
## (Intercept) 5.8215724 6.2992404
## Distance -2.5566110 -2.3497072
## TimeD -0.9372951 -0.6174832
## VisFull -1.6684553 -0.9929307
## Distance:TimeD 0.5307503 0.8233565
## Distance:VisFull 0.8560806 1.1486868
## TimeD:VisFull 0.3791876 0.8314700
## Distance:TimeD:VisFull -0.8675873 -0.4537797## R2m R2c
## 0.2955039 0.4907164There is a significant 3-way interaction between distance, timepoint, and vis-type (highlighted orange). Following the same procedures as before, we ran the same model on the 24hr and 48hr time points, separately.
Post-Hoc: 15 annotated
| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 6.060406 | 0.1249065 | 48.519560 | 0 |
| Distance | -2.453159 | 0.0561902 | -43.658145 | 0 |
| VisFull | -1.330693 | 0.1766444 | -7.533174 | 0 |
| Distance:VisFull | 1.002384 | 0.0794649 | 12.614166 | 0 |
Post-Hoc: 15 annotated
| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 4.7297134 | 0.1273115 | 37.1507207 | 0.0000000 |
| Distance | -1.4507754 | 0.0534162 | -27.1598429 | 0.0000000 |
| Time48 | -0.1720603 | 0.0825656 | -2.0839223 | 0.0371672 |
| Distance:Time48 | 0.0163699 | 0.0755419 | 0.2166996 | 0.8284424 |
Post-Hoc: 63 track
| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 6.0604064 | 0.1161452 | 52.179558 | 0 |
| Distance | -2.4531591 | 0.0521412 | -47.048379 | 0 |
| Time48 | -0.7773891 | 0.0805949 | -9.645642 | 0 |
| Distance:Time48 | 0.6770534 | 0.0737388 | 9.181780 | 0 |
The analysis of the 24hr time point revealed a significant 2-way interaction between distance and vis-type (highlighted in orange). To break this down, we recentered distance around the farthest time point (1.78).
Post-Hoc: 24hr re-centering
| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 1.693783 | 0.1235493 | 13.709376 | 0.0000000 |
| DistanceCenter1.78 | -2.453159 | 0.0561902 | -43.658145 | 0.0000000 |
| VisFull | 0.453550 | 0.1747250 | 2.595793 | 0.0094373 |
| DistanceCenter1.78:VisFull | 1.002384 | 0.0794649 | 12.614166 | 0.0000000 |
This analysis revealed that there was still an effect of vis-type (highlighted in blue) but in the opposite direction. Meaning that, at the center of the storm, participants believed that the 15 track annotated display would receive less damage than the 63 track display. But at the farthest time point (1.78) participants rated the 15 track annotated display as receiving significantly more damage.
Post-Hoc: 48hr
| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 5.2830172 | 0.1204081 | 43.875930 | 0.00e+00 |
| Distance | -1.7761057 | 0.0488170 | -36.382903 | 0.00e+00 |
| VisFull | -0.7253642 | 0.1702828 | -4.259763 | 2.05e-05 |
| Distance:VisFull | 0.3417002 | 0.0690377 | 4.949471 | 7.00e-07 |
Similar to the 24hr timepoint, there was a significant interaction between distance and vis-type (orange).
Post-Hoc: 48hr re-centering
| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 2.1215491 | 0.1193465 | 17.7763762 | 0.0000000 |
| DistanceCenter1.78 | -1.7761057 | 0.0488170 | -36.3829028 | 0.0000000 |
| VisFull | -0.1171379 | 0.1687815 | -0.6940207 | 0.4876693 |
| DistanceCenter1.78:VisFull | 0.3417002 | 0.0690377 | 4.9494705 | 0.0000007 |
When we recentered the 48hr time point around the farthest probe location, there was no longer a significant difference between the 15 track annotated display and the 63 track display (also highlighted in blue).
Confidence
Confidence: 7, 15, 63
Vis: 7 is the referant| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 4.8158750 | 0.1884513 | 25.555013 | 0.0000000 |
| Distance | -0.0182110 | 0.0164066 | -1.109977 | 0.2670089 |
| TimeD | -0.1137417 | 0.0193626 | -5.874303 | 0.0000000 |
| Vis15 | 0.3386324 | 0.2639974 | 1.282711 | 0.1995933 |
| Vis63 | 0.6277500 | 0.2653011 | 2.366180 | 0.0179727 |
Confidence: 15 annotated vs. 15 unannotated
| Estimate | Std..Error | t.value | p.z | |
|---|---|---|---|---|
| (Intercept) | 5.1302977 | 0.1839973 | 27.8824567 | 0.0000000 |
| Distance | 0.0596917 | 0.0193643 | 3.0825713 | 0.0020522 |
| Time | -0.0028981 | 0.0009522 | -3.0435447 | 0.0023381 |
| VisFull | 0.0503676 | 0.2556802 | 0.1969947 | 0.8438316 |
Follow-up Questions
1. We are very interested in how you made your decisions about the level of damage that the oil platform would incur. Please describe in as much detail as possible your decision-making process. This will greatly help us learn about how people make decisions with hurricane visualizations. 2. Do you have experience with hurricane forecasts?
| Forecasts | length(unique(ResponseID)) |
|---|---|
| No | 190 |
| Yes | 11 |
3. Have you lived or do you live in an area that experiences hurricane threats?
| Threats | length(unique(ResponseID)) |
|---|---|
| No | 172 |
| Yes | 29 |
4. The display shows the hurricane getting large over time.
| Vis | Larger | length(unique(ResponseID)) |
|---|---|---|
| 15 | False | 36 |
| 15 | True | 15 |
| 63 | False | 23 |
| 63 | True | 27 |
| 7 | False | 26 |
| 7 | True | 24 |
| Full | False | 20 |
| Full | True | 30 |
Doing a follow-up regression analysis using the 15 track annotated as the referant, we find that participants viewing the 15 track annotated visualization were more likely to believe that the display showed the hurricane getting larger over time than those viewing the 15 track unannotated display.
| Estimate | Std..Error | z.value | Pr…z.. | |
|---|---|---|---|---|
| (Intercept) | 0.4054651 | 0.2886751 | 1.4045723 | 0.1601485 |
| Vis15 | -1.2809338 | 0.4216369 | -3.0380023 | 0.0023815 |
| Vis63 | -0.2451225 | 0.4047822 | -0.6055663 | 0.5448028 |
| Vis7 | -0.4855078 | 0.4043038 | -1.2008491 | 0.2298097 |
5. The display indicates that the forecasters are less certain about the path of the hurricane as time passes.
| Vis | Variable | length(unique(ResponseID)) |
|---|---|---|
| 15 | False | 27 |
| 15 | True | 24 |
| 63 | False | 24 |
| 63 | True | 26 |
| 7 | False | 21 |
| 7 | True | 29 |
| Full | False | 26 |
| Full | True | 24 |
6. Areas on the map not covered by the visualization will not be hit by the hurricane.
| Vis | Hit | length(unique(ResponseID)) |
|---|---|---|
| 15 | False | 35 |
| 15 | True | 16 |
| 63 | False | 32 |
| 63 | True | 18 |
| 7 | False | 35 |
| 7 | True | 15 |
| Full | False | 36 |
| Full | True | 14 |
7. Did you recognize the areas in the map?
| Recognize | length(unique(ResponseID)) |
|---|---|
| No | 19 |
| Yes | 182 |