RED Digital Cinema’s most recent patent granted two weeks ago, titled Video Image Data Processing In Electronic Devices, joins the other three RED patents related to compressed raw [8,872,933, 9,479,749, 9,787,878]. However, this one seems more general and tries to circulate various sensors and devices. Does RED want to expand its raw compression capabilities to other devices and sensors?
RED’s new patent related to compressed raw
The patent was granted on November 15. 2022 no. 11,503,294 named Video Image Data Processing In Electronic Devices, was filed by Jim Jannard (RED Digital Cinema founder), Jarred Land (RED’s CEO), and Manjunath Subray Baht. The patent presents new methods for compressed raw, that in our opinion, are more general and comprehensive, and related to smartphones, digital cameras, and other devices and sensors (including Foveon). The patent includes familiar schemes like the RED Hydrogen and DSMC2 bodies. Here’re the highlights below:
Background of the Patent
Here’s the background for the patent titled Video Image Data Processing In Electronic Devices: “High-end video capture capability continues to advance. Professional cinematographers as well as increasingly large numbers of consumers are demanding high-quality video recording capability, not only in traditional digital video cameras, but also in other video camera-equipped electronic devices including cellphones/smartphones, tablets, and the like”. The next part of the patent is presenting the definition of compressed raw.
Definition of compressed raw
The patent states and defines that the raw image data can be “raw” in the sense that the video data is not “developed”, such that certain image processing image development steps are not performed on the image data prior to compression and storage. Such steps can include one or more of interpolations (for example, de-Bayering or other de-mosaicing), color processing, tonal processing, white balance, and gamma correction. For example, the compressed raw image data can be one or more of mosaiced (for example, not color interpolated, not demosaiced), not color processed, not tonally processed, not white balanced, and not gamma corrected. Rather, such steps can be deferred for after storage, such as for off-board post-processing, thereby preserving creative flexibility instead of than “baking in” particular processing decisions in the camera.
General compression methodologies
According to the application, these image processing and compression techniques can be implemented in a variety of form factors. For instance, the techniques described herein for compressing and on-board storage of compressed raw image data can be implemented in a relatively small-form-factor device, such as a smartphone having an integrated camera (or multiple cameras including front camera(s) and rear camera(s), or a small form factor camera. For instance, the processing techniques according to certain embodiments are tailored for implementation in a small form factor device having a relatively limited power budget, processing capability, and physical real estate for incorporation of electronic components, etc. In another example, the compression techniques described herein can be implemented in relatively larger form factor cameras, including digital cinema cameras. Moreover, the patent mentions that although the electronic devices described may be primarily written in the context of a smartphone, the disclosures are applicable to any of a variety of electronic devices with or without cellphone functionality, including digital still and motion cameras, personal navigation devices, mobile internet devices, handheld game consoles, or devices having any or a combination of the functions or other functions.
Discrete cosine transform (DCT)
The patent refers to a method called DCT, by explaining that the electronics residing in the image capture device can be configured to, as part of the compression, transform the raw mosaiced image data using the DCT (discrete cosine transform) method, or another transform (such as a transform that defines a finite sequence of data points in terms of a sum of functions oscillating at different frequencies) to obtain transform coefficients, and compress the transform coefficients. According to some embodiments, the electronics can be configured to perform the compression without using an image frame memory (for example, a dynamic random access memory [DRAM]) that stores a full image frame for processing purposes. For instance, the electronics can compress the transform coefficients using an on-chip first memory (for example, a static random-access memory [SRAM]) that is integrated with an image processing chip (for example, an application-specific integrated circuit [ASIC] or field-programmable gate array (FPGA]), and without using any second DRAM or other memory positioned off-chip.
Facilitating memory-depended compression to enhance power efficiency
This part explains the method of facilitation that elevates power efficiency in order to allow the raw-compression in mobile devices. As stated: “In certain embodiments, the electronics can nonetheless include a DRAM or other second memory off-chip. However, the off-chip memory in such embodiments may be used for purposes other than compression of raw video image data, such as for pixel defect correction, addressing pixel pattern noise, or the like. This is unlike existing image capture devices, such as smartphones, which use an off-chip DRAM to perform image compression. For instance, some existing image capture devices use an off-chip DRAM to calculate motion vectors for H.264 compression. Certain embodiments described herein use DCT techniques, thereby facilitating memory-efficient compression, without the need to calculate motion vectors or use off-chip memory. Therefore, performing compression without the use of a full image frame memory (for example, an off-chip DRAM) enhances power efficiency (such as, by around 0.5 Watts (W) in some implementations), which is particularly useful in a small-form-factor device such as a smartphone. According to certain aspects, the electronics of the image capture device consume less than 15 W or less than about 20 W during operation”.
Resolution of 20K and beyond
According to the patent application, depending on the embodiment (for example, depending on sensor size) the techniques for on-board storage of compressed raw image data described herein can be capable of achieving resolution levels of at least 2 k, 3 k, 4 k, 4.5 k, 5 k, 6 k, 8 k, 10 k, 12 k, 15 k, 20 k, or greater resolution levels.
Even relevant to the Foveon sensor
The application emphasizes that the image sensor used for the described compression method can be any type of video sensing device, including, for example, but without limitation, CCD, CMOS, vertically-stacked CMOS devices such as the Foveon sensor, or a multi-sensor array using a prism to divide light between the sensors.
Summary
The current raw compression patents owned by RED Digital Cinema run till 2028. However, this one will last until 2037. Thus, expect a solid amount of lawsuits initiated by other camera manufacturers who are eager to implement internal compressed raw capabilities in their cameras. Meanwhile, we’re waiting for a resolution regarding the ongoing mitigation and legal process of the RED – Nikon case. Stay tuned, as we’ll keep you posted. The question remains: Does RED want to expand its raw compression capabilities to other devices and sensors?
Yossy et al,
This is an interesting topic (thanks for putting these pieces together Yossy!), but I am only replying to one question ():
RED wants to prevent competition, as simple as that. It is all about control.
RED has tried to enter the mobile market and failed, no matter how interesting that would have might been, but quite the same, that market is under control and was -unfortunately- impossible to crack, at least with the capabilities of Jim at that time and other potential constraints given when Hydrogen was intended to enter the market.
Will be interesting to see what happens in that area, since the whole „internal compressed raw recording“ issue seems too trivial to even be patented, but somehow RED managed not only to get the patent, but even defend it.
This patent should have been tied to a very specific codec (REDs codec) and not like a blueprint for all compressed raw recordings – doing that internally is only a matter of processing power and granting that patent was clearly a lack of understanding that that is simply a natural technical advancement.
If this patent application is granted again, it will only prevent progress.
But that is what control often is about, isn‘t it ?
Fun fact: Did you know that a housewife patented the separate use of 2 computers, one for work and one for private use ?
On chip, probably even in memory image compression has been done for years. So, what has it to offer that’s unique, new and novel?
Indeed – not much, except RAW, which together with J2k was probably relatively early and innovative, but not necessarily exclusive (why many were so quickly successful with decoding it in the early days). Cineform did probably something similar, except as an external company, doing software only, was never patented alongside with a particular hardware integration.
I think conform was patented, Red uses the technology.
The jp2K still at so many frames a second version wasn’t really innovative, as it’s a prior innovation, and using it at a burst rate (witch is what filming actually is) was discussed publically many times and done.
The question was somehow stripped from my answer, it was from your article:
“Does RED want to expand … to other devices and sensors?“
DCT? Really? It´s around since DV (yes the tape). And ist is the foundation of jpg compression. How on earth can ist be part of a red Patent? Scratching Head…
Red’s patents are pretty solid. If you can beat Apple in court, you’ve definitely got a case. I do hate that technological progress is currently being held back by Red. Compressed raw would have become an industry standard across all cameras & platforms had Apple won their Pro Res Raw case. But you have to give props where they’re due. Red beat everyone to compressed raw by years, not months. They invested a ton of money building their company from nothing and compressed raw was always at the center of it. We don’t have to like it (and I really, really don’t) but Red earned their right to hold the patents to compressed raw. Camera companies just need to get creative the way 3rd party companies reverse engineered PC’s in the 80’s. They need to find the loopholes. Blackmagic did it. And there are others out there. Companies just need to find them.
When have they ever Won on court? They got the problem to go away before court with Apple. I don’t think that patent has ever fern tested in a full court process.
The problem is, that the Red related entities would have a range of patents. They might be in play in negotiations.
I would be glad to sea this worked out.
When have they ever Won on court? They got the problem to go away before court with Apple. I don’t think that patent has ever fern tested in a full court process.
The problem is, that the Red related entities would have a range of patents. They might be in play in negotiations.
I would be glad to sea this worked out.
What exactly are they patenting which is new? This article is convoluted, and I’m to sick to read again. It needs to be down to the point. A lot of stuff mentioned is old prior technology which isn’t patentable. What new stuff are the trying to patent. Is it some type of pre processing, or an attempt to extend the original patent decades longer?
Patent is granted and here:
https://patents.google.com/patent/US11503294B2/en
Nothing here feels remotely in any way novel or new. This is all 30 year old technology. RED razzle dazzles an incompetent patent office & steals from humanity. Fuck RED so much. What a gaggle of fuckwit asschildren RED is. Evil evil bastards.
The original RED RAW patent was going to expire in a few years, so the filing of a related patent to essentially extend the basic coverage doesn’t seem that surprising.
Doing it in the area of mobile could be an interesting risk, RED don’t have the kind of deep pockets of the big mobile players.
While Apple didn’t “succeed” in fighting RED’s RAW patent, it should be remembered that ProRes RAW is an insignificant atom in the ocean of Apple’s business – not worth Apple’s time.
The moment RED actually get in Apple’s way in a significant matter related to iPhones, Apple have to funds to battle RED in court until RED have no resources left.
Let’s get realistic. Did Apple fail at all in court, did anybody? Has these patents being tested?
Let’s be realistic, Red has a lot of hidden patents. You turn up at a meeting with the goods to take on this patent, and Red can turn up with a lot of other parents and argue you are in violating or need to licensed. Then, a business decision has to be made. It should be made illegal to tie or threaten with any other matter, pattern or license in negotiations and agreements to do with patent litigation, in order to stop potential practices like that. Then, maybe a lot more situations would be settled.
The law should have a community requested review of patent process, where a group of companies, licensees, or other people in the community, as a group, can request a review with reasons of a patent. Then the free process of review starts in depth review if it is truely novel, how novel, finding prior art and community usage and knowledge of things in the patent, and finding collusion in agreements. With evidence freely submitted before during and after, and evidence automatically triggering a pre-community review (a pro community review) without it needing to be requested. Then, on the basis of areas in a patent struck out, limited, or the whole patent being limited, the licensor is required to repay any licence paid no that part. If deliberately done, or after finding out the part is invalid, then from that point the licensor is responsible for all associated costs. It is regarded as a serious criminal matter. That there be a voluntary restitution process in the patent office, where upon funding out there is some sort of invalidity to a patent, the licensor can apply to come in to the process and reimburse licencees, with minium cost. That the patent office perform reviews of patent, and when they suspect it might be an violation.
That the patent office is a powerhouse of economic growth and the balance of trade, and therefore not be required to run at a profit, but be required to spend as much money as reasonably necessary to make sure the process is done right and make justice.
To make it a criminal offence to knowingly try to circumnavigate, control or license anything on a patent which is not patentable.
Then, maybe a lot of these concerns will disappear, as everybody is motivated to make sure these things aren’t in violation. The government can vote on it and pass legislation, this year.