Large-scale SDXL study

Attack-invariant projection experiment

Separate learned metrics for frozen DINO image embeddings and frozen caption embeddings. Conclusions are opened only on untouched prompts and held-out attack families.

large_scale_attack_invariant_v1

Track conclusions

caption_blip_base

not supported

Distance ratio
2.437
95% prompt CI
[2.101, 2.819]
PCA ratio
2.785
Paired log gain
-0.029
Paired 95% CI
[-0.081, 0.022]
Recall@1
0.394
Triplet accuracy
0.667
dinov2_base

not supported

Distance ratio
23.436
95% prompt CI
[19.146, 28.701]
PCA ratio
22.672
Paired log gain
0.009
Paired 95% CI
[-0.100, 0.117]
Recall@1
0.746
Triplet accuracy
0.951
dinov3_l16

supported

Distance ratio
18.312
95% prompt CI
[15.794, 21.014]
PCA ratio
7.252
Paired log gain
1.545
Paired 95% CI
[1.364, 1.728]
Recall@1
0.754
Triplet accuracy
0.932

Learned and PCA 3D views

dinov2_base learned

point 0point 1point 2point 3point 4point 5point 6point 7point 8point 9point 10point 11point 12point 13point 14point 15point 16point 17point 18point 19point 20point 21point 22point 23point 24point 25point 26point 27point 28point 29point 30point 31point 32point 33point 34point 35point 36point 37point 38point 39point 40point 41point 42point 43point 44point 45point 46point 47point 48point 49point 50point 51point 52point 53point 54point 55point 56point 57point 58point 59point 60point 61point 62point 63point 64point 65point 66point 67point 68point 69point 70point 71point 72point 73point 74point 75point 76point 77point 78point 79point 80point 81point 82point 83point 84point 85point 86point 87point 88point 89point 90point 91point 92point 93point 94point 95point 96point 97point 98point 99point 100point 101point 102point 103point 104point 105point 106point 107point 108point 109point 110point 111point 112point 113point 114point 115point 116point 117point 118point 119point 120point 121point 122point 123point 124point 125point 126point 127point 128point 129point 130point 131point 132point 133point 134point 135point 136point 137point 138point 139point 140point 141point 142point 143point 144point 145point 146point 147point 148point 149point 150point 151point 152point 153point 154point 155point 156point 157point 158point 159point 160point 161point 162point 163point 164point 165point 166point 167point 168point 169point 170point 171point 172point 173point 174point 175point 176point 177point 178point 179point 180point 181point 182point 183point 184point 185point 186point 187point 188point 189point 190point 191point 192point 193point 194point 195point 196point 197point 198point 199point 200point 201point 202point 203point 204point 205point 206point 207point 208point 209point 210point 211point 212point 213point 214point 215point 216point 217point 218point 219point 220point 221point 222point 223point 224point 225point 226point 227point 228point 229point 230point 231point 232point 233point 234point 235point 236point 237point 238point 239point 240point 241point 242point 243point 244point 245point 246point 247point 248point 249point 250point 251point 252point 253point 254point 255point 256point 257point 258point 259point 260point 261point 262point 263point 264point 265point 266point 267point 268point 269point 270point 271point 272point 273point 274point 275point 276point 277point 278point 279point 280point 281point 282point 283point 284point 285point 286point 287point 288point 289point 290point 291point 292point 293point 294point 295point 296point 297point 298point 299point 300point 301point 302point 303point 304point 305point 306point 307point 308point 309point 310point 311point 312point 313point 314point 315point 316point 317point 318point 319point 320point 321point 322point 323point 324point 325point 326point 327point 328point 329point 330point 331point 332point 333point 334point 335point 336point 337point 338point 339point 340point 341point 342point 343point 344point 345point 346point 347point 348point 349point 350point 351point 352point 353point 354point 355point 356point 357point 358point 359point 360point 361point 362point 363point 364point 365point 366point 367point 368point 369point 370point 371point 372point 373point 374point 375point 376point 377point 378point 379point 380point 381point 382point 383point 384point 385point 386point 387point 388point 389point 390point 391point 392point 393point 394point 395point 396point 397point 398point 399point 400point 401point 402point 403point 404point 405point 406point 407point 408point 409point 410point 411point 412point 413point 414point 415point 416point 417point 418point 419point 420point 421point 422point 423point 424point 425point 426point 427point 428point 429point 430point 431point 432point 433point 434point 435point 436point 437point 438point 439point 440point 441point 442point 443point 444point 445point 446point 447point 448point 449point 450point 451point 452point 453point 454point 455point 456point 457point 458point 459point 460point 461point 462point 463point 464point 465point 466point 467point 468point 469point 470point 471point 472point 473point 474point 475point 476point 477point 478point 479point 480point 481point 482point 483point 484point 485point 486point 487point 488point 489point 490point 491point 492point 493point 494point 495point 496point 497point 498point 499

dinov2_base PCA

point 0point 1point 2point 3point 4point 5point 6point 7point 8point 9point 10point 11point 12point 13point 14point 15point 16point 17point 18point 19point 20point 21point 22point 23point 24point 25point 26point 27point 28point 29point 30point 31point 32point 33point 34point 35point 36point 37point 38point 39point 40point 41point 42point 43point 44point 45point 46point 47point 48point 49point 50point 51point 52point 53point 54point 55point 56point 57point 58point 59point 60point 61point 62point 63point 64point 65point 66point 67point 68point 69point 70point 71point 72point 73point 74point 75point 76point 77point 78point 79point 80point 81point 82point 83point 84point 85point 86point 87point 88point 89point 90point 91point 92point 93point 94point 95point 96point 97point 98point 99point 100point 101point 102point 103point 104point 105point 106point 107point 108point 109point 110point 111point 112point 113point 114point 115point 116point 117point 118point 119point 120point 121point 122point 123point 124point 125point 126point 127point 128point 129point 130point 131point 132point 133point 134point 135point 136point 137point 138point 139point 140point 141point 142point 143point 144point 145point 146point 147point 148point 149point 150point 151point 152point 153point 154point 155point 156point 157point 158point 159point 160point 161point 162point 163point 164point 165point 166point 167point 168point 169point 170point 171point 172point 173point 174point 175point 176point 177point 178point 179point 180point 181point 182point 183point 184point 185point 186point 187point 188point 189point 190point 191point 192point 193point 194point 195point 196point 197point 198point 199point 200point 201point 202point 203point 204point 205point 206point 207point 208point 209point 210point 211point 212point 213point 214point 215point 216point 217point 218point 219point 220point 221point 222point 223point 224point 225point 226point 227point 228point 229point 230point 231point 232point 233point 234point 235point 236point 237point 238point 239point 240point 241point 242point 243point 244point 245point 246point 247point 248point 249point 250point 251point 252point 253point 254point 255point 256point 257point 258point 259point 260point 261point 262point 263point 264point 265point 266point 267point 268point 269point 270point 271point 272point 273point 274point 275point 276point 277point 278point 279point 280point 281point 282point 283point 284point 285point 286point 287point 288point 289point 290point 291point 292point 293point 294point 295point 296point 297point 298point 299point 300point 301point 302point 303point 304point 305point 306point 307point 308point 309point 310point 311point 312point 313point 314point 315point 316point 317point 318point 319point 320point 321point 322point 323point 324point 325point 326point 327point 328point 329point 330point 331point 332point 333point 334point 335point 336point 337point 338point 339point 340point 341point 342point 343point 344point 345point 346point 347point 348point 349point 350point 351point 352point 353point 354point 355point 356point 357point 358point 359point 360point 361point 362point 363point 364point 365point 366point 367point 368point 369point 370point 371point 372point 373point 374point 375point 376point 377point 378point 379point 380point 381point 382point 383point 384point 385point 386point 387point 388point 389point 390point 391point 392point 393point 394point 395point 396point 397point 398point 399point 400point 401point 402point 403point 404point 405point 406point 407point 408point 409point 410point 411point 412point 413point 414point 415point 416point 417point 418point 419point 420point 421point 422point 423point 424point 425point 426point 427point 428point 429point 430point 431point 432point 433point 434point 435point 436point 437point 438point 439point 440point 441point 442point 443point 444point 445point 446point 447point 448point 449point 450point 451point 452point 453point 454point 455point 456point 457point 458point 459point 460point 461point 462point 463point 464point 465point 466point 467point 468point 469point 470point 471point 472point 473point 474point 475point 476point 477point 478point 479point 480point 481point 482point 483point 484point 485point 486point 487point 488point 489point 490point 491point 492point 493point 494point 495point 496point 497point 498point 499

dinov3_l16 learned

point 0point 1point 2point 3point 4point 5point 6point 7point 8point 9point 10point 11point 12point 13point 14point 15point 16point 17point 18point 19point 20point 21point 22point 23point 24point 25point 26point 27point 28point 29point 30point 31point 32point 33point 34point 35point 36point 37point 38point 39point 40point 41point 42point 43point 44point 45point 46point 47point 48point 49point 50point 51point 52point 53point 54point 55point 56point 57point 58point 59point 60point 61point 62point 63point 64point 65point 66point 67point 68point 69point 70point 71point 72point 73point 74point 75point 76point 77point 78point 79point 80point 81point 82point 83point 84point 85point 86point 87point 88point 89point 90point 91point 92point 93point 94point 95point 96point 97point 98point 99point 100point 101point 102point 103point 104point 105point 106point 107point 108point 109point 110point 111point 112point 113point 114point 115point 116point 117point 118point 119point 120point 121point 122point 123point 124point 125point 126point 127point 128point 129point 130point 131point 132point 133point 134point 135point 136point 137point 138point 139point 140point 141point 142point 143point 144point 145point 146point 147point 148point 149point 150point 151point 152point 153point 154point 155point 156point 157point 158point 159point 160point 161point 162point 163point 164point 165point 166point 167point 168point 169point 170point 171point 172point 173point 174point 175point 176point 177point 178point 179point 180point 181point 182point 183point 184point 185point 186point 187point 188point 189point 190point 191point 192point 193point 194point 195point 196point 197point 198point 199point 200point 201point 202point 203point 204point 205point 206point 207point 208point 209point 210point 211point 212point 213point 214point 215point 216point 217point 218point 219point 220point 221point 222point 223point 224point 225point 226point 227point 228point 229point 230point 231point 232point 233point 234point 235point 236point 237point 238point 239point 240point 241point 242point 243point 244point 245point 246point 247point 248point 249point 250point 251point 252point 253point 254point 255point 256point 257point 258point 259point 260point 261point 262point 263point 264point 265point 266point 267point 268point 269point 270point 271point 272point 273point 274point 275point 276point 277point 278point 279point 280point 281point 282point 283point 284point 285point 286point 287point 288point 289point 290point 291point 292point 293point 294point 295point 296point 297point 298point 299point 300point 301point 302point 303point 304point 305point 306point 307point 308point 309point 310point 311point 312point 313point 314point 315point 316point 317point 318point 319point 320point 321point 322point 323point 324point 325point 326point 327point 328point 329point 330point 331point 332point 333point 334point 335point 336point 337point 338point 339point 340point 341point 342point 343point 344point 345point 346point 347point 348point 349point 350point 351point 352point 353point 354point 355point 356point 357point 358point 359point 360point 361point 362point 363point 364point 365point 366point 367point 368point 369point 370point 371point 372point 373point 374point 375point 376point 377point 378point 379point 380point 381point 382point 383point 384point 385point 386point 387point 388point 389point 390point 391point 392point 393point 394point 395point 396point 397point 398point 399point 400point 401point 402point 403point 404point 405point 406point 407point 408point 409point 410point 411point 412point 413point 414point 415point 416point 417point 418point 419point 420point 421point 422point 423point 424point 425point 426point 427point 428point 429point 430point 431point 432point 433point 434point 435point 436point 437point 438point 439point 440point 441point 442point 443point 444point 445point 446point 447point 448point 449point 450point 451point 452point 453point 454point 455point 456point 457point 458point 459point 460point 461point 462point 463point 464point 465point 466point 467point 468point 469point 470point 471point 472point 473point 474point 475point 476point 477point 478point 479point 480point 481point 482point 483point 484point 485point 486point 487point 488point 489point 490point 491point 492point 493point 494point 495point 496point 497point 498point 499

dinov3_l16 PCA

point 0point 1point 2point 3point 4point 5point 6point 7point 8point 9point 10point 11point 12point 13point 14point 15point 16point 17point 18point 19point 20point 21point 22point 23point 24point 25point 26point 27point 28point 29point 30point 31point 32point 33point 34point 35point 36point 37point 38point 39point 40point 41point 42point 43point 44point 45point 46point 47point 48point 49point 50point 51point 52point 53point 54point 55point 56point 57point 58point 59point 60point 61point 62point 63point 64point 65point 66point 67point 68point 69point 70point 71point 72point 73point 74point 75point 76point 77point 78point 79point 80point 81point 82point 83point 84point 85point 86point 87point 88point 89point 90point 91point 92point 93point 94point 95point 96point 97point 98point 99point 100point 101point 102point 103point 104point 105point 106point 107point 108point 109point 110point 111point 112point 113point 114point 115point 116point 117point 118point 119point 120point 121point 122point 123point 124point 125point 126point 127point 128point 129point 130point 131point 132point 133point 134point 135point 136point 137point 138point 139point 140point 141point 142point 143point 144point 145point 146point 147point 148point 149point 150point 151point 152point 153point 154point 155point 156point 157point 158point 159point 160point 161point 162point 163point 164point 165point 166point 167point 168point 169point 170point 171point 172point 173point 174point 175point 176point 177point 178point 179point 180point 181point 182point 183point 184point 185point 186point 187point 188point 189point 190point 191point 192point 193point 194point 195point 196point 197point 198point 199point 200point 201point 202point 203point 204point 205point 206point 207point 208point 209point 210point 211point 212point 213point 214point 215point 216point 217point 218point 219point 220point 221point 222point 223point 224point 225point 226point 227point 228point 229point 230point 231point 232point 233point 234point 235point 236point 237point 238point 239point 240point 241point 242point 243point 244point 245point 246point 247point 248point 249point 250point 251point 252point 253point 254point 255point 256point 257point 258point 259point 260point 261point 262point 263point 264point 265point 266point 267point 268point 269point 270point 271point 272point 273point 274point 275point 276point 277point 278point 279point 280point 281point 282point 283point 284point 285point 286point 287point 288point 289point 290point 291point 292point 293point 294point 295point 296point 297point 298point 299point 300point 301point 302point 303point 304point 305point 306point 307point 308point 309point 310point 311point 312point 313point 314point 315point 316point 317point 318point 319point 320point 321point 322point 323point 324point 325point 326point 327point 328point 329point 330point 331point 332point 333point 334point 335point 336point 337point 338point 339point 340point 341point 342point 343point 344point 345point 346point 347point 348point 349point 350point 351point 352point 353point 354point 355point 356point 357point 358point 359point 360point 361point 362point 363point 364point 365point 366point 367point 368point 369point 370point 371point 372point 373point 374point 375point 376point 377point 378point 379point 380point 381point 382point 383point 384point 385point 386point 387point 388point 389point 390point 391point 392point 393point 394point 395point 396point 397point 398point 399point 400point 401point 402point 403point 404point 405point 406point 407point 408point 409point 410point 411point 412point 413point 414point 415point 416point 417point 418point 419point 420point 421point 422point 423point 424point 425point 426point 427point 428point 429point 430point 431point 432point 433point 434point 435point 436point 437point 438point 439point 440point 441point 442point 443point 444point 445point 446point 447point 448point 449point 450point 451point 452point 453point 454point 455point 456point 457point 458point 459point 460point 461point 462point 463point 464point 465point 466point 467point 468point 469point 470point 471point 472point 473point 474point 475point 476point 477point 478point 479point 480point 481point 482point 483point 484point 485point 486point 487point 488point 489point 490point 491point 492point 493point 494point 495point 496point 497point 498point 499

Held-out breakdowns

TrackGroupValueRatio
dinov2_baseattack_familyocclusion_overlay23.726
dinov2_baseattack_familysocial_media_compound52.402
dinov2_baseseveritymedium38.608
dinov2_baseseveritymild202.008
dinov2_baseseveritysevere12.977
dinov2_baseprompt_sourcecoco201723.546
dinov2_baseprompt_sourcediffusiondb23.326
dinov2_basecaption_divergencehigh26.522
dinov2_basecaption_divergencelow16.841
dinov2_basecaption_divergencemedium26.944
dinov3_l16attack_familyocclusion_overlay36.725
dinov3_l16attack_familysocial_media_compound20.299
dinov3_l16severitymedium38.211
dinov3_l16severitymild134.723
dinov3_l16severitysevere16.239
dinov3_l16prompt_sourcecoco201714.926
dinov3_l16prompt_sourcediffusiondb21.697
dinov3_l16caption_divergencehigh24.385
dinov3_l16caption_divergencelow12.882
dinov3_l16caption_divergencemedium17.668
caption_blip_baseattack_familyocclusion_overlay18674430583332.262
caption_blip_baseattack_familysocial_media_compound209386037637.075
caption_blip_baseseveritymedium33521335307699.848
caption_blip_baseseveritymild57554702363810.000
caption_blip_baseseveritysevere13334414092827.037
caption_blip_baseprompt_sourcecoco20172.493
caption_blip_baseprompt_sourcediffusiondb2.382
caption_blip_basecaption_divergencehigh2.766
caption_blip_basecaption_divergencelow2.210
caption_blip_basecaption_divergencemedium2.336

Representative retrieval geometry

prompt_000002

occlusion_overlay

Original source image
Original source
Attacked derivative
Attacked derivative
Different SDXL seed
Different seed
prompt_000006

occlusion_overlay

Original source image
Original source
Attacked derivative
Attacked derivative
Different SDXL seed
Different seed
prompt_000007

occlusion_overlay

Original source image
Original source
Attacked derivative
Attacked derivative
Different SDXL seed
Different seed

Caption limitations

Exact collision rate: 0.741. Empty rate: 0.000.

Image/caption semantic-distance Spearman correlations:

Artifacts and provenance