Ai poses an enormous problem ~ it's much more than image scanning and Copyright issues

[updated 30 March, 2023 - see the end of the post]

The question of Copyright and AI scanners can thundering home recently when I learned that my Flickr images have been included in at least two major AI datasets. So, I submitted the following Help Request to Flick.com - 

Dear Sirs, 

Using tools available to see if my images have been consumed by "AI" scanners I see that, yes, indeed, my Flickr images have been scanned. 

I am very aware of Copyright and it's limitations. I used to advise legal council at the companies I worked for in technology. I did NOT give anyone the right to scan and then use my works. 

I hold copyrights to all but one of my images, and I believe there may be a serious issue by allowing "AI" scanners to do what they are doing. 

What is Flickr's position on this? 

 

Indeed, among may other tasks, I helped advise the lawyers at the last company I worked for.  I'm fairly familiar with what is and is not allowed, and where the gray areas are in copyright law and its application.  Perhaps Flickr had developed a position on the topic?

Their reply, from what I can tell, is likely a lawyer reviewed/sanitized/canned message that gets sent to everyone who asks about AI -

"Hi there,

Thank you for your input on this matter.

As this is a new and emerging space, we have not yet fully reviewed how Flickr will fit with AI images and photography. 

At this time, there have not been any changes to the Copyright Act to address AI generated images.

It is something we are reviewing closely and should there be any changes that would affect the whole platform, we will certainly notify our members.

As always, if there are instances where a creator identifies their copyright or license has taken place, they can submit a claim through our DMCA process for review and actions here.

Best,
Doug

We're updating the terms of Flickr's free accounts to strengthen our community and the long-term stability of Flickr. Read more here."

 

First, I fail to see where there is something unique about AI that could require legal redefinition/reapplication.  Secondly, Flickr recently announced a new area of AI generated images where people can post the output of AI software.

It feels as if Flickr is giving this a rather big "pass."

Class Action lawsuits have been filed and I wondered how to join. One site I visited indicated that _normally_ 

"...you don't need to do anything to "join" a class action. If your legal rights are affected by a class action, you usually will only need to get involved once the case settles. In most cases, you will need to submit a claim, either online or through the mail, to receive your portion of the settlement or judgment..." 

Digging into Flickr's past I found that four years ago it was known that Flickr had been scrapped by an AI in a project that involved IBM. 

"...“This is the dirty little secret of AI training sets. Researchers often just grab whatever images are available in the wild,” said NYU School of Law professor Jason Schultz..." 

One more thing. There was an article published back in 2018 on scrapping Flickr for "deep learning experiments", complete with code to implement a scrapper where it was noted that - 

"...1. It is not legal and ethical to scrape some websites 

2. Even if you were not concerned about the law or ethics, scrapping at scale can be challenging particularly if there are safeguards in place to discourage scrapping..." 

 

Which begs the question: Did Flickr allow scrappers/scanners, or did they just not care and let it happen?  Afterall, they'd published an API that people could with as they pleased.

Any way I look at it I'm wondering if the benefit stills photographers in the "Pro" program _pay_ for out weighs the huge issue of creative copyrighted works  AI poses and has posed for several years. Flickr may on some level be culpable.

I follow a number of tech websites to try and keep current with the State of Things in science and technology.  Something very concerning turned up just the other day (March 24, 2023 to be exact).

After listening to the podcast I came to the understanding and realization that the problems posed by AI far out-strip the scanning of our images.  

AI is out of control and, if not contained, might pose serious risks to us. As this fire is building to disturbingly large proportions, Microsoft has laid off key personnel from their AI Ethics team.  Isn't _now_ the time to staff-up ethics teams on AI?  What the h*ll is going on?

Ack!  Something very important is happening right this moment and I've not been following the ethical and moral implications of AI as closely as I could have.

If you're at all interested in this topic, have a listen. 

[Update 30 March, 2023] - Some people are calling for an outright ban on AI.

 

Yes,the images used to illustrate this blog entry
were carefully chosen to set the mood

One last Wabbit Hole ~ and it's a whopping BIG one, too!

A little "accidental Renaissance"
to keep things interesting in prep for
reading through somewhat dense material.

 

[References section updated twice ~ 4 March, and 3 March, 2023]

After whinging and whining about the lack of verifiable, accurate knowledge, I find myself at the bottom of an Enormously Vasty Wabbit Hole.  At the top and just as I fell in I saw it was labeled  "Knowledge You Are Looking For."

The areas I wanted to learn more about but was having a Devil of A Time finding anything useful/correct/truthful included dynamic range, sensor noise, and color depth. I was still poking at the things that might go into defining what a "Fat Pixel" camera might be made up of.

Apologies first: For this blog entry I'm going to move very fast and rather deep.

There are nuances and details that can and should be applied to each of the following statements.  Each step can be an entire study unto itself.  As always, don't trust me, but if you must, make sure you verify.

Here is my present state of understanding.  

Sensor Basics ~

~ There is an analog input side of every sensor ending at analog to digital converters (ADC) with the following components -

• Light sensitive photo site
• Analog amplifier
• ADC
  

~ There is a digital output side of every sensor starting at the ADC with the following components -

• Digital image processing chips for stills and video
• Digital image processing software for stills and video
  

~ ISO controls the gain on the _analog_ amplifier that feeds the ADC

~ RAW files are made up of the digitized data spit out of the ADC

~ jpg files are the result of the ADC output _plus_ whatever massaging the manufacturer applies on the digital circuitry in-camera

~ In-camera jpg processing _may_ further increase gain digitally (as a second gain function)

Dynamic Range ~

This is the EV difference between highlight areas with detail and shadow areas with detail.  Sensor noise reduces dynamic range.  Said another way, the quieter the sensor the greater the dynamic range.

In general...

~ The broadest dynamic range is seen at the Base ISO

~ The higher the gain on the analog amplifier (increasing ISO) the lower the dynamic range

Sources of noise ~

~ The analog amplifier (where signal gain is first applied) that feeds the ADC and ADC itself are the primary sources of noise in RAW files (see Canon DSLRs)

~ In-camera jpg processing digital circuit may further increase gain and thereby add noise downstream from the analog circuit.  In general, if there is noise on the output of the ADC, unless there is noise reduction on the digital side, we will see noise out of the digital circuits, too (see Canon DSLRs).

Sensors ~ ISO Variant

These are the traditional CMOS sensors that we've all come to know and love.  Until very recently, these were the only ones commercially available to us.

~ The electron gathering well at each sensor site has these properties -

• Traditional big and somewhat shallow electron gathering well design
• When used at low ISO settings (ie: low analog amplifier voltages) we get -
  
• Broad dynamic range
  
• Slight pixel to pixel variations (subtle noise, if you will)
• Lowest noise/broadest dynamic range tells us what the Base ISO is
• As ISO increases -
  
• Dynamic range decreases
  
• Noise increases... maybe... (see astro-photography reference video below about cases where this is _not_ entirely true)
  

 Recent Sony Design Enhancements ~ ISO invariance

Sensors starting with A6300, A7RIII, A7III, and very obviously the A7SIII, as well as the Sony manufactured MF sensors used in Fuji GFX cameras have _two_ electron gathering wells per photo-sensor site.

~ First electron gathering well at each sensor site has these properties -

• Traditional big and somewhat shallow well design
• Used at low ISO settings
• Broad dynamic range
  
• Slight image to image variations (subtle noise, if you will)
• Lowest noise again tells us what the Base ISO is
  

~ Second electron gathering well at each sensor site has the following properties -

• Narrow but deep well design
• Used for higher ISO settings
• Shows reduced dynamic range
• BUT it shows _less_ noise than the big, shallow electron well
• Highest dynamic range when using this higher ISO than base ISO 1 well-type sets a Second Base ISO
   

Interesting property: ISO invariant sensors tend to show no change in noise as ISO is increased after the Second Base ISO has been switched on.

Note: There are examples of sensors that are ISO invariant from low ISO, such as the Nikon D750 where noise levels do not change from ISO200 on up. See the astro-photography reference video below for the Nikon example.

Comments ~

 Taken as a whole I feel I can begin to understand a few seemingly unrelated things.  Such as -

• Early Kodak CCD MF sensors showed the best image quality at the time.  This, even if they _expanded_/_amplified_ the ADC output from 14bits to 16bits.  At least they started out at 14bits when everyone else was down around 10bits of RAW and/or stuck at 8bits jpg.  I can see where the "Fat Pixel" ideas could stem from in terms of perceived image quality.  What I've learned is that modern sensors can easily outperform the original Kodak MF sensors in every measurable way, _and_ we get ISO flexibility with current sensors where Kodak worked best at Base ISO, period.
  
• I now see (on Photons to Photos - see reference link below) the differences between the old 14bit ADC Canon 5D MkII and 7D sensors and the _12bit_ ADC Sony A6000.  In one release of products we went from "ya, that's not bad but if we're not careful there's loads of noise in the shadows" to "wow! now there's a clean image".  The A6000 has more dynamic range at 12bits ADC and, as hoped for, less noise than the older 14bit ADC Canon sensors.  There is likely something about Canon's analog circuit that was rather prone to the introduction of error.  It's interesting to me that it's only with the release of Canon's new mirrorless cameras that their sensors seem to meet or slightly exceed the early Sony FF sensors.  This tells us something about how much work Canon has done on the analog side photo-site to ADC path.
  
• Looking (again on Photons to Photos - see reference link below) at first generation Sony A7, A7S, and A7R, there's not really all that much difference between these models in terms of dynamic range and noise.  Yes, the A7S has slightly better dynamic range than the A7, but I'm not sure I'll ever actually see or appreciate the extra 1/3stop EV the A7S has.  I've read where we can begin to see differences of 1EV dynamic range, but I haven't verified that for myself.  Coming back to the Photons to Photos information, the A7R looks nearly as good as the A7S.  In the end, if I can't make a great "Fat Pixel" image with any of these three cameras, I'm doing something drastically and dramatically wrong.
• ETTR (Expose To The Right) "works", but not for the reasons we're commonly told.  It "works" because we are able to bring the dark areas up out of the base level noise.  Of course we have to guard against saturating the highlights beyond recovery, but once the data has been collected, it's more flexible in processing than images where the dark areas are down in the noise base of the sensor.
  
• ETTL (Expose The The Left) "works" as expected.  We use ETTL in black and white photography as a "lazy man's way" of guaranteeing as much detail in the highlights as possible.  This can closely emulate film images when processed appropriately.  The highlights are raised in processing and the shadows raise with the highlights. As an aside, I've started to use Zebras to know when the highlights are saturated.  Coming back to ETTL, with quiet sensors we might be able to avoid distracting noise to the shadow areas.  Which leads me directly to the next item.
  
• One of the problems I had trouble understanding why there is so much noise in the shadow areas of severely under-exposed images even at 100ISO where I thought we should see the lowest noise everywhere across a broad dynamic range image.  It turns out the noise is easily explained.  The electron gathering wells aren't able to gather enough information consistently across the wells to make the dark areas appear as smooth as the light areas.  The light simply is not available at those low levels to distribute evenly. Hence noise, even at 100ISO.  To avoid this problem in a single shot, use ETTR or better yet Zebras.  If the dynamic range of the scene is broader than a sensor can handle, there's HDR and image stacking during processing.
  
• Potential recent "Fat Pixel" candidates could be the Sony sensor manufactured Fuji MF GFX cameras.  Their dynamic range exceeds by 1EV the best Sony FF.  The MF Fujis are well above my current Pay Grade and I doubt we'll ever see them go for less than 1500USD used.  Yes, I know.  The 50R is trading hands on the used market for around 2250USD.  It's still too rich for me.  But there is something to be said about well-engineered sensor development and ISO invariant circuitry.  Kodak sensors were never ever close to being this good.
• I now understand why backside illumination of a sensor "works."  By raising the black base to a known level, sensor noise levels are suppressed by starting at zero (pure black) well above the potentially noise inducing analog circuits.  It's a neat trick, actually.  I feel there are some creative solutions being applied to sensor design these days.  This is Fun Stuff.
  
• Pursuing the absolute best color depth, longest dynamic range, and lowest noise images possible requires a static subject, a tripod, setting the camera at its Base ISO 1, and shooting three or four identical images.  Three image stacked low ISO photos "work" because the process averages out the subtle fat/shallow electron well variations.  The final output image quality should easily exceed that of, well, just about anything.  And speaking of which...
  
• Sony has come up  with something interesting in their dual Base ISO sensors.  While destined for video work, I can see where there will be benefits for us stills shooters, too.  Even if we're stuck at 8EV dynamic range at Base ISO 2, I'm still struck by the possibilities of lower than Base ISO 1 noise.  

It's a Crazy Topsy Turvy Counter-Intuitive World out there and I can't wait to see what the Sensor Development Wizards come up with next.

Until further notice I have stopped reading the thoughts, evaluations, and conclusions of the vast majority of popular still-photography related websites.  In general they are at a distinct lack for meaningful data on sensor performance, dynamic range, color depth, and noise characteristics.  I know they're "trying" the best they can, but I've found that I have a difficult (Gear Grinding) time with the Marginal at Best "test" methods and bad data they attempt to analyze and justify their conclusions with.  It's become just "noise" to me.

 

Sometimes a person has to dig deeper to get at truth.

 

References ~ [updated/modified on 4 March (correcting a link to the most important video, the first, and updated slightly on 3 March, 2023]

Here is a video with very clear, concise explanations of Sony sensor behaviors.  

I had to pause the video every other sentence, or so it seemed, so I could stop and think about and think through what was just said.  There is so much _good_ information here.  In fact, _this_ should be the reference for discussion of sensor performance.  I wish more of YouTube was this accurate and informative.  I've found there is too much "squishy thinking" go'n on out there!!!  So this is an absolute breath of fresh air.  

It's easy to see how much this video influenced the organization of my thoughts.  While I knew various pieces and parts of the process, the video helped me organize what I knew into a cohesive whole.  In fact, I've borrowed from the video outline in the writing of this piece because I found it that useful.

Photons to Photos has a great site on measured dynamic range, noise, and sensor performance. Complete with methods and rationale explaining what they do and why. For some reason I've not run into this site until now.  There were many spent hours looking around, reading, and comparing various sensor data.  It's amazing how much a person can learn if they're patient and aware and take a few notes along the way.

Here is a video on ISO invariance and why it's interesting and useful.  There are also comparison images showing how ISO variant and invariant sensors behave differently. 

I often wondered what changed with the introduction of Sony's cameras around 2012.  Reading Jon Rista's explanation from 19 January 2015 - 01:30 PM gives a clue.  While I might not agree that 500nm waffer technology was a problem (except as an example of Canon not keeping up with the latest waffer fabrication trends), the rest of his argument rings true.  Further, Jon Rista gives us a few potential clues about "Fat Pixel" photo-sites from Posted 29 January 2015 - 12:09 AM (pay attention to his comments on photo-site area and electron well capacities). For a Geek who really wants to Geek Out it's very interesting and potentially practical stuff

Lastly, here is a link to a Wiki page where we can see a Sony camera and function matrix.  In addition, Sony has a matrix of ADC bit depth information.  This too is organized by camera and function.  Using this information we can begin to guess how much circuitry is implemented in the various cameras that influence things like read speed and camera capabilities.

To me the most important outcome of all the Geeky Nerdiness is the acquisition of knowledge that can be applied and balanced in the real world when pursuing highest possible image quality.

Update of a common 'net expression from when dinosaurs roamed the earth: Base ISO 1, f/8 and be there.  

If it's dark out, Base ISO 2, wide open and be there.