Target is the thing that is being debugged (e.g., Javascript, page, service worker)
It may be weird that you have two separate list... some are one 1 to 1.
but others allow a 1 to many relationship. Like web pages can debug many things
Frontend is our UI.
And backend is the page we are inspecting>
Protocol is used to communicate between "front end" and "backend"
Going to focus this talk about the protocol... the protocol is auto generated.
There is also the "remote", which is important for devices... like connecting to iOS and Simulator
With remote, we get some assurances around IPC
Front end is just vanilla javascript and HTML, CSS, etc.
The InspectorFrontendHost is just a special API (non-standard)
And InspectorFrontendAPI is how things talk to web inspector
We use a few frameworks to help us (e.g., codeMirror).
We use extensive use of event listeners
We have a "custom layout engine"TM
we use this as a "view", and we use requestAnimationFrame etc to keep everything in sync.
And we follow a simple MVC
Controllers are mainly "managers"... mostly a singletons on the frontend
They handle the protocol communication
They need to know which target they are communicating with and handling
But other than that is just models and views
The models are just data wrappers
The UI is split into multiple areas that interact with each other
As you interact, each area is given metadata to control various aspects of web inspector.
Web inspector care about RTL, Dark mode, eetc. so you need to care about that when contributing to the project
There are various APIs attatched to WI.* ... WI.Table, WI.NavigationBar, etc. as components.
Lots of examples of how these components are used. Makes it easier for new developers to contribute
The protocol is almost entirely auto generated
Basic IPC system based on JSON
There are a ton of protocols
And there are other domains.
Like CSS domain corresponds to CSS things, runtime is "javascript"
new domains are just specified with JSON
beyond domains, there are corresponding types
It provides type safety and assurances
Plus it provides events and expected structures
Devin shows an example of what a definition of a domain looks like in JSON
Similarly with types... we define it as "type": "integer "
Makes it really easy to understand... same with enums
e.g., "before" and "after"
and gets more complex from there
but not too complex
You can also specify "commands"... like., "getDocument"... and that would send from frontend to backend, and it sends a serialized version of the command.
As a developer, you just implement the the frontend call, backend function, but you don't need to worry about the IPC. That's all handled for you. (edited)
And for events, there is just parameter associated with it. It keeps things simple.
Web inspector needs to consider:
Inspector Web Process
Inspector UIProcess
Inspected UIProcess
Inspected WebProcess
and how each communicates / route through each one. We handle that for you. You never have to touch this code to make it easier.
But what's important is that "inspector" and "inspected" are totally different devices potentially. But you don't need to worry about that as a developer
It's possible to inspect from a new macOs to an older iOS, but older iOS can't inspect a newer macOS... i.e., you can inspect "the past", but not the future
We have automated systems to check for support if the backend has support for a particular command
Which is very helpful. So you can recover
Again, this is all auto generated,
....Devin shows how this is done with JS... communication between front and backend
... lots of JS code...
Similarly with events... it's not that hard
Heavy use of WTFJSON
Backend... each debugable has a top level controller object
It's responsible to create an agent to help with a domain... it provides the functionality for that domain.
E.g., all would have a runtime agent because they all talk "javascfript", but they are normally specialized
There are specialized objects to help you work with different things you might want to debug.
General tips... a lot of prior out throughout the code base. Only confusing bit is the protocol.
But generally everything is pretty straight forward
You can even inspect web inspector with web inspector
We rely on ESLint
to keep the code clean and consistent
There are barely no UI tests... you don't have to write tests.... but the downside is that you need to manually test stuff
No need to wait for EWS... it's easy to make changes to the UI
Razvan, when you create agents on the backend, how do you maintain sprawl?
generally try to think to figure out what category it falls into? (e.g., look on MDN)... you could argue that you could make a new domain. Try to use an existing agent if you can. Or if it requires a special case that talks to multiple agents. You need to think hard about the domain and the agents to handle different cases. So yeah, think about the architecture and reuse as much as possible
I tried to make changes to the code UI... do you plan to add UI frameworks?
we are not huge fans of UI frameworks because they can get in the way. We like to name things well, and keep things super simple. The view system is our answer to that. We have a very basic system, it's only like 20 lines of code.
We define the rules. And we enforce them on ourselves
The less code we have to deal with the better. Nowadays, we don't need things like jQuery anymore because how much better browsers are now
Don from Sony. You are allowed to use whatever is available in the web platform. What web platform would you like to see more of?
we see a lot uses of var but generally we leave those alone if it's working. We use new things as a test bed, but we don't got out of our way. Like, going forward, we always replace functionality old functionality and also use new things when they become available?
do you use ESLint on EWS or plan to?
more often than not, we use it as guide but we sometimes don't agree with ESLint for various reasons. So, more of a guide an principle.
comment: Karl. I use web inspector a lot for web compat. It helps a lot when you fix web inspector it helps fix web platform health overall.
Please fix more stuff
Devin: sure. We want to help everyone - web developers and WebKit engineers
We sometimes even do very specific things for web inspector developers. And we add things specific things just for WebKit engineers, but also benefit web developers
Frances. What suggestions do you have to find Web Inspector bugs for webkit engineers?
here is no way to explicitly find them, but there is no specific list. I'm sure there are quite a few
Razvan: to rephrase is bugzilla, the right place to file Webkit bugs?
yes, put things under the Web Inspector components
Elika, great presentation! Where can we find this info later?
Great question. I want to write a blog post about this. I don't have speaker notes.
Bundled in WebKit for Apple WebKit pots.. Integration with GStreamer Encoders, Hardware accelerated support requiring custom decoder/sinks
2019 onwards: New GstWebRTC announced at GSTConf
2017, and the initial experimentation which we upstreamed a backend for in 2022
Screen capture when you want to capture mic or video. In WebKit that takes a permission from UIProcess to WebProcess
We differ a bit form the apple ports in that I think apple does it in the GPU process
we had to design from RPC from the webkit process to the ui process
that means it’s not easily testable right now
we have some infra for this, it breaks now and then, it’s not that hard to fix but you have to watch it
one pipeline per capture device
Stream capture (getDisplayMedia()) - we have two layers of permissions
in linux we have the webkit permission request + desktop portal - we have to talk to it over IPC
The desktop portal can remember previous permission requests so perhaps we can improve that
then we have another pipeline which relies on pipewire, it’s deplohyed on linux desktops nowadays and it can captire - it provides DMABufs
for MediaStream handling - when you need to capture, in the gstreamer ports we have a plugin that is able to feed the popline with data coming from the media stream
we use that quite a lot for handing incoming streams, canvas/video/capture
{shows a diagram showing a simplified version}
webrtcbin implements the JavaScript SEssion Establishment Protocol, the integration with webkit was quite easy - GstPromis is hand in hand with webkit’s Promise
When you receive a media stream, in javascript you will usually connect that to a video element (as .src) - in webcore it will internally hook to webkitmediastreamsrc -
this all means that we use the same thing for normal playback of video, etc — which I think is different form apple ports again
additional APIs: Can’t do everything that is required of the spec, some we did upstream adds — we had to do some plumbing to provide frames-encoded, bitrate, etc
also with DataChannel - we had to do some patches in Gstreamer, but now it is integrated quite well
we are passing ~60% of the tests, but that isn’t WPT
there is still work to do obviously, but we’re getting there
we’re also doing dedicated bring ups of specific platforms - we focuesed on gaming with amazon luna - we followed kind of the chrome approach, I tried with a safari UA but it doesn’t work.. So we have to send the chromium UA and rely on some legacy RTC…
we have a pull request, it’s not been reviewed yet
we had some issues with the service expecting some chrome specific specs - we provided some shim implementations of those to just allow us to move forward - it’s not great
another one we’re bringing up is jitsi — it’s unfortunately not working yet.
it requires features we don’t yet support: renegotiation, simulcast, etc
Takeaway: Every platform requires a dedicated bring up. We can have spec compliance, but every one requires more… Being spec compliant is just the tip of the iceberg for webrtc
re: encoding - we have to provide for simulcast - GSTWebRTC has some support but it’s not used by WebKit yet…
there are 2 types of scalability
re: Privacy and security… All of our pipelines run in the WebProcess, we can’t send that into production, that’s bad
ideally the plan on the roadmap is to move that to a restricted NetworkPRocess
etc
another deals with capture devices - same issue in the WebProcess… tentative plan is to improve permissions/portal on GTK/Desktop. We’ll have to figure it out on embedded where user input could be lacking. Then we’d like to move all of the GStreamer pipelines to the GPU process
How do I accelerate video? are there licensing issues?
on adb platforms the support is provided by third party provider GCMA plugin that you can use so it really depends on each case
second question: You mentioned the keyboard for cloud gaming. Did you have any plan to upstream that implementation?
I dont think there are positive signal from apple at this point. WE could provide an implementation but it would have to be disabled by default because there were some issues related with that spec that we’ve not addressed yet
Indeed
The question was about keyboard-lock, for the record
The more we use RenderLayout, the more things we get for free (z-index, etc.)
LBSE was designed as a drop-in replacement for the old SVG engine
This wasn’t well-received; too much change too quickly
So we had to rename the entire legacy engine so it could live in parallel with LBSE
Last we checked, the LBSE patch is about 40MB
We provide a compile-time flag allowing to pick between engines
Issue was opened in 2021, hadn’t been finished yet
Overall progress status is around 77%
The main thing missing is resources
Many patches landed since last year’s WKCM
November 2022, we landed a patch that unified geometry and transform computations with CSS
The key for performance is to bypass WebCore
Access already-composited images and move them instead.
This let us finally switch on accelerated transforms, this year
That patch landed June 2023
We still had repainting issues; if you animated inside an SVG with CSS, looking at it Web Inspector caused a lot of repaints
This was patched out October 2023
We started to upstream resources in September 2023
In SVG, resources are clipping, masks, filters, etc.
In our next steps, we want to finish upstreaming, including many patches adding missing layout tests
Resource invalidation is broken in WebKit since forever
(shows example, slide 20/32)
How invalidations actually work: if you change a circle radius to 20, the clipper has to notify the masker that the rect it’s using has changed
Before, we handled this via layout
So changes to radius would force a re-render of the mask
Do it in the right order, and you could get lucky with correct invalidation
Get them out of order, and invalidation wouldn’t happen properly
We suffered a lot from unnecessary repaints and relayouting as a result of this
(shows example, slide 22/32)
Here’s a typical code graph
At some point, we’ll tell core the layout is dirty
We have to invalidate it
Afterwards, something is marking it for invalidation. Why?
At some point in this recursive calling, the element gets a style change notification
We have to mark the user of that clip path and “needs layout”
Resources in the chain are notified
If you process in the right order but swap elements, then after layout, things are still dirty and so layout isn’t finished when we think it is
We think we can avoid this mess if we stop abusing layout
All this is a fundamental flaw we need to avoid
Invalidation depends on element order, and it should not
Our plan: LBSE will do masking and clipping statelessly
No more temporary buffers or caching of individual render objects
This is being implemented via RenderSVGResourceContainer, a new base class, which landed this month (Oct 2023)
The next target after that is the resource clipper
We’re currently rewriting the whole of the resource invalidation logic
The SVG resource cache will be moved to LegacySVGResourcesCache
Mid-term plans: finish support for , then add , linear and radial gradients, , and
All of those become trivial after finishing with
First basic implementations should be done this year
Long-term plans: finish LBSE, such that all layout tests pass and are not slow
We have to reduce the core RenderLayer overhead and cache more SVG subtree information
Also selectively construct render layers only when necessary
Target for switching over LBSE by default and remove the legacy engine is 2024
Igalia is working with all the power we can, getting everything we have upstream so we get everything upstream though maybe not as performant as legacy
Then we will work with Apple to reach performance targets and equal or exceed the legacy engine in performance
MotionMark is a good example of something that doesn’t benefit from the new engine
On other tests, LBSE is outperforming the legacy engine by orders of magnitude
Jen Simmons gave an overview of Masonry layout at WWDC.
Grid: consists of rows and columns. This might leave a lot of space in individual cells.
Masonry: solves this by getting rid of rows.
[Shows example of 7 sibling elements where the first three elements are next to each other on the first conceptual row. The elements after that are put in the shortest subsequent conceptual rows.]
Worked on updating the specification to finalize it. Worked on Align Tracks & Justify Tracks and because there was a lot of complexity and a11y issues these were removed.
Intrinsic Track Sizing Changes: we used to use the first conceptual row to determine the sizes. That made it hard for web developers to use.
In the new specification all elements participate in determining the track size.
recent contributor to web inspector. I have a lot experience as a front end engineer
Today I'm going to be presenting a "all fonts" feature for the web.
Frances makes a joke about bad font usage.
Some fonts are not appropriate for.
The New York Time uses a particular font to convey their identity.
I created a site with different font for each word.
How can a web dev find out which font is being used for each word?
You can do that by inspecting each element?
You can look at font-family under style... but you can't know which one has been actually selected. It's unclear.
And then you have Times New Roman, but it's a default so it's still confusing.
You can also look at the CSS itself to see what is being applied being ids
and via classes
Your last option is to go to Firefox and their dev tools. But that still doesn't tell you which one WebKit chose.
Proposals: what is you had a way of seeing which font is selected via a new fonts tab.
Challenges: what if you have a slide show, for instance, and each slide is using a different font, or scripts are changing things, and what if the computed style is different for different devices?
Right now, I've managed to get the right information to show in console.log().
there is a repo on GitHub that WebKit uses to be asked and provide feedback for web standards
intended to be separate from actual implementation
sometimes have to implement things WebKit doesn't like
ideally allows for evaluation of standards with a more principled/conceptual viewpoint
collaboration can be async
to request a position, discuss with colleagues, post a comment asking, give a deadline, and (usually) WebKit will do something before then
there's also a basic website that visualizes the state of the standards position repo on GitHub with some extra features (e.g. search, filtering, etc.)
would love to have more contributors. often only Apple employees
could use help with triaging of applying labels to issues. some is automated, but not all. all contributors should be able to do so
Q: opinions from real use cases often are missing from standards positions
A: more of this would definitely be useful. there are a few cases where this has happened, but not often. could be that the web developers themselves arent involved. WebKit should encourage them to
Q: are there any examples where there's disagreement on standards positions? there might be a little bit of hesitancy around "internal" disagreement stopping momentum of standards work
A: there are a few issues with disagreement from the wider web, but not often from within WebKit. possibly the mailing list used to have more "internal" disagreement. there is the #standards/#standards-github slack channels for less public discussion if you'd prefer to "poll the room" or verify opinions or etc. before posting publicly
Wasm is a target language for compiling web programs. It’s an alternative to JS where you want to transpile
it’s been used for a number of interesting applications - bringing photoshop to the web, things that use plugins written this way - flight simulator, games
you might wonder: is that low level stuff all it can be used for?
wasm - can it be used by language that use GC?
Kludgy solutions, but Key missing piece was is some kind of allocatable memory - that’s what this GC proposal is for
it has datatypes like structs and arrays
and also new kinds of reference types - you get new type casts and advanced types from this
at least in the browser context it is designed to take advantage of the built in already
(shows a concrete example with type declaration)
(and globals initialized with new types)
(and writing operations which access that data)
(and new reference types)
there are a bunch more features, ut this is a high level overview of what it is
let’s talk about the implementation progress in JSC
most of the features of the proposals are already implemented
but it’s not in a shippable state - a few more months — bulk array operations aren’t implemented, misc missing instructions — it needs much more testing and optimization but look how much (a lot) is already implemented
Basically: It’s well underway and hopefully we can get it to shipping soon.
The big takeaway is that this is an exciting time for web assembly — this proposal enables WASM to be used for a whole bunch more languages. It’s stage 4 pretty much the last stage — it’s shipped in other browsers
it enables targeting wasm for Java, OCaml and many others
It's important to get APIs right on the web the first time — you can't unship something. Once web developers start using it, it stays
There's no version 2 of the web. "there's web, and there's more web". It's an additive platform
"Web 2.0 isn't really a thing. There's web, and then there's more web." Classic
Consider developer experience when designing API. Is it easy / intuitive? How consistent is it with other APIs? How does it interact with other APIs? Does the API have unexpected behavior for end users?
Providing feedback on WIP (or complete) specs is really helpful for spec editors — please provide feedback
As implementors, this is part of the job, not just translating spec to code
When building web API, building demos (which are different than testcases) is helpful. Show how developers will actually use the API. Show what the end user experience would be like.
This helps communicate the value of what you're building
Building demos help discover flaws in your API design
Step 2: Test your implementation
Web platform tests are great and should be part of the process, but you should determine if they match the spec, and whether they are exhaustive. Are all cases covered?
Example: for layout, there are sometimes lots of edge cases not covered, like putting something in too small of a container
Add a feature flag
See UnifiedWebPreferences.yaml
These are recently overhauled, with documentation available to help guide usage
Measure the quality of your implementation
WPTs
Interop
Performance and power
How well do your demos work?
Test in STP, not just minibrowser. Things like Autofill can introduce surprising interactions not present in Minibrowser
Test on iOS (and work with Apple if needed)
Ensure spec + implementations are aligned. If implementations are not aligned, and nobody speaks up, users and developers suffer
Test your demo
You should strive to ship three things:
Feature
Tests
Spec changes / improvements
How to toggle feature flags in Safari?
Enable debug menu (docs online to do this)
Go to Feature Flags section, should find list of flags with toggles
Some possible follow-ups after your feature ships:
Demos are not only useful for guiding your feature development, but also for evangelism and helping teach developers about new web technology. You can ping @Jen Simmons or @jond if you have something you feel is worth showing
Q: Is there a place where we can put such demos on webkit.org?
A: webkit.org/demos. Also often embedded into webpages / posts / release articles. Reach out to Jen or Jon for help
Q: What if the spec is under incubation? Would it be good to start upstreaming implementation? i.e. the spec isn't ready yet, changes are still happening
A: It depends. How much is the spec expected to change?
As a concrete example, CSS grid went through many iterations. There was an initial implementation from Microsoft that helped guide improvements. Once spec eventually stabilized, more implementors came on board
Having the early implementation was really helpful in this case
Different implementors might interpret specs differently, which can help expose ambiguity, ultimately creating a stronger web platform for everyone (edited)
A lot of things on the web are broken for many reasons, causing a negative user experience. This can be due to websites/libraries targeting a specific browser, a new feature, an old feature being removed, a bug fix that websites rely on, or tracking prevention.
These problems can be rectified with UA overrides (lying about who you are, e.g., modifying the user agent or HTTP header), C++ quirks (working at code-level implementation), and hot-fixes on websites.
The first browser to push website hotfixes was Opera BrowserJS (pre-Blink) by modifying JS/CSS content after it loaded.
Source/WebCode/page/Quirks.cpp is a mostly true/false file which is based on domain and simple heuristics. The actual Quirks logic is elsewhere in the C++ codebase.
Example of a quirk in practice: ikea.com experienced an issue where images at the bottom of the page would not load after a WebKit change. WebKit pushed a quirk that performed a specific, different behavior for that specific website. Ikea subsequently patched the issue, at which point the quirk was removed.
Quirk implemented by @Brandon at: https://github.com/WebKit/WebKit/pull/6595
Quirk removed by @Karl Dubost at: https://github.com/WebKit/WebKit/pull/14058
For UA overrides: HTTP User-Agent + navigator.userAgent.