Graphics¶

This document describes how the web contents are painted into the final output in GTK and WPE ports of WebKit. The goal is to explain how things currently work, however to understand some of the concepts it's important to know a bit of history and how we ended up with our current model.

Non-accelerated single process model: At the beginning the graphics model was quite simple, all web contents were rendered into a graphics context in a single process.
Multi-process model: Then, the multi-process model was introduced (WebKit2) and the rendering model was more or less the same but in a web process. So, we had to find a way to send the rendered contents from the web process to the UI process. The web process rendered into a Cairo image surface and the resulting pixels were copied to a shared memory buffer that the UI process read to create a Cairo surface to pass to the GTK widget.
Accelerated compositing: The web became more and more complex with animations and other features that required an accelerated compositing model to work properly. The contents were split into layers that were rendered with Cairo image surfaces and then upload as textures with OpenGL. The TextureMapper was introduced to compose all those textures and produce a composited frame. On every paint request, all the layers were flushed, rendered and then composited in the main thread. The shared memory approach to share the rendered contents with the UI process was no longer possible, since the output is now not on main memory but on GPU, and copying the pixels from GPU to main memory for every frame would be very slow. We added a redirected X composite window, used as the target surface of the GL context where the frames were composited and the XDamage extension to notify the UI process when the window was updated. On the UI process a Cairo X11 surface was used to render the generated XPixmap into the GTK widget.
Wayland: The way to share composited frames with the UI process was X11 specific, so when Wayland support was introduced we had to find a different way. For Wayland we added a nested compositor running in the UI process. The web process connected to the nested compositor to create a surface that was used as the target surface of the GL context where the frames were composited. In the UI process the Wayland surface was rendered into the GTK widget by using EGL images when GDK supported it, or downloading the texture to a Cairo image surface otherwise.
Threaded compositor: Composition was moved to a dedicated thread to release the main thread while compositing layers. The challenge here was to coordinate the layer changes between the main and composited threads. We reused the CoordinatedGraphics implementation that Qt port was using at the time to do the composition in the UI process, adapting it to our threaded model. Performance improved a lot, but the code became increasingly complex.
WPE port: The WPE port was upstreamed, with initial focus on Wayland, and later adding APIs to allow applications to handle rendering in both the web and UI processes.
Threaded rendering: Compositing was already in a dedicated thread, but layer flush and rendering was still happening in the main thread. With the idea of releasing more the main thread, the painting was also moved to different threads. The composition still happens after the layer flush, but it waits for all rendering threads to complete before composing the layers.
Async scrolling: Scrolling was also moved to a different thread when possible to ensure responsiveness. Scroll events are handled by the scrolling thread triggering a composition if needed without having to go through the main thread.
DMA-BUF: The GTK port started using DMA-BUF buffers for the composited frames. In this case composition is rendered into a surfaceless context and buffers are allocated with GBM. This allowed us to remove all the code to create GL context from X11 and Wayland targets. WPE also adopted this approach for its new WPE platform API.
Display link: For the synchronization with the actual screen we relied on the X11 server or Wayland compositor, notifying the web process when current frame has already been presented on screen to allow the next one to start. A Display link implementation was added to notify the web process on every vblank signal we get from the screen when animations are running.
Skia: Cairo was replaced by Skia as the 2D rendering library used to paint the layer contents. This allowed us to use the GPU also to render the layers.

UI process rendering¶

The UI process receives the composited frames from the web process to be presented on the final output. Once the frame is painted the UI process notifies the web process. The way in which the composited frame is received and painted depends on every port.

Rendering frames in WebKitGTK¶

The GTK port supports two rendering modes: the accelerated and non-accelerated mode. In non-accelerated mode, the web process produces frames by sharing an image using shared memory buffers. Whenever a new frame is available the web process sends the Update message to the UI process including the file descriptor of the shared memory containing the bitmap and a list of updated rectangles. The updated rectangles are then painted into a Skia image and a web view redraw is scheduled to paint the image into the web view widget. In accelerated mode, the GTK port always uses the DMA-BUF buffers to get the composited frames from the web process. AcceleratedBackingStoreDMABuf is the class that handles all accelerated drawing in GTK port. When the web page is created the set of supported DMA-BUF buffer formats and the DRM device to be used to allocate buffers is sent to the web process. The web process tries to find the first combination of format and modifier from the list that is supported by the driver. When the web process allocates a new buffer the message DidCreateBuffer is sent to AcceleratedBackingStoreDMABuf with an id and all the parameters needed to import the buffer. Depending on the GTK version and GDK options the buffer is imported differently, when GTK version is recent enough and GDK is using EGL or vulkan, the buffer is imported using GdkDmabufTextureBuilder. The import happens only once, when the buffer is created. When a composited frame is ready, the message Frame is sent to the UI process, just with the id of the buffer, since it has already been imported, and a fence file descriptor. The buffer is stored as pending and the fence is monitored so that when it's triggered, a repaint is scheduled on the web view widget. When the web view is then painted, a new GdkTexture is created with gdk_dmabuf_texture_builder_build, the previous committed buffer is released (a ReleaseBuffer message is sent to the web process to let it know that buffer can be used again) and the pending one is set as committed. The new GdkTexture is passed to GTK to be painted in the widget and the FrameDone message is sent to the web process to notify that the buffer has already been painted.

Rendering frames in WPE new API¶

WPE also uses the DMA-BUF renderer when using the new API. As in the GTK port, the list of supported DMA-BUF buffer formats and the DRM device to be used to allocate buffers is sent to the web process when the page is created, but in the case of WPE the format negotiation is dynamic. For example in the Wayland platform, we receive a notification that preferred buffer formats have changed when the surface becomes a candidate for direct scanout (typically the surface is in fullscreen mode with no transparencies). In that case a new list of buffer formats and devices is built and sent to the web process with the message PreferredBufferFormatsDidChange. When the next frame is going to be rendered in the web process, the old buffers are destroyed and a new one is allocated for the format and modifiers that better match using the given device. This will make it possible for the Wayland compositor to pass the generated DMA-BUF buffer directly to the screen without any composition. When the web process allocates a new buffer the message DidCreateBuffer is sent to AcceleratedBackingStoreDMABuf with an id and all the parameters needed to import the buffer and a WPEBufferDmaBuf is created. When a composited frame is ready, the message Frame is sent to the UI process, just with the id of the buffer and a fence file descriptor. If the WPEDisplay supports explicit sync, the fence file descriptor is set on the WPEBufferDmaBuf to be handled by the platform and the buffer is passed to the display to be rendered, otherwise the fence is monitored by polling the file descriptor like in the GTK port and the buffer is rendered once the fence is triggered. The WPE platform then renders the buffer and notifies back when it's done. At this moment the pending buffer becomes the committed one and FrameDone message is sent to the web process. The platform also notifies when a buffer can be released, optionally including a fence, so that a ReleaseBuffer message is sent to the web process passing the buffer id and the optional fence file descriptor. The new WPE is not limited to DMA-BUF buffers but at the moment they are the only ones supported.

Rendering frames in WPE old API¶

WPE uses libwpe for frame rendering, so depending on the backend the rendering will be done differently. libwpe is used from both the UI and web processes and it uses its own way to communicate between both processes. When the web process is created a new renderer host client is created with wpe_renderer_host_create_client() which returns a file descriptor that is passed to the web process. The library name of the current libwpe backend is also passed to the web process. When the web process is initialized the backend is loaded passing the given library name to wpe_loader_init() and a global PlatformDisplay is created for the given render host client file descriptor. The PlatformDisplay creates an EGL render backend using wpe_renderer_backend_egl_create() that receives the render host file descriptor. Then the EGL display is initialized for the platform and native display provided by the EGL render backend. Now every web view needs also a way to communicate with the web process to get its contents rendered. In the old API web views are created with a mandatory construct parameter to provide the wpe view backend. When a web page is created wpe_view_backend_get_renderer_host_fd() is called to get a file descriptor to be passed to the web process. On the web process side, the WebKit compositor creates a surface that creates an EGL render backend target calling wpe_renderer_backend_egl_target_create() that receives the render host file descriptor. Then an EGL context is created for the native window returned by wpe_renderer_backend_egl_target_get_native_window(). For every frame the compositor notifies the EGL render backend target before and after the composition by calling wpe_renderer_backend_egl_target_frame_will_render() and wpe_renderer_backend_egl_target_frame_rendered(). The composition rendering happens directly in the native window provided the WPE backend, so buffer handling depends on the backend and it's all handled there.

Web process¶

The web process renders the page contents producing a rendered frame to be consumed by the UI process to be displayed into the output. The non-accelerated mode is quite simple, whenever a repaint is requested the drawing area schedules a display using a timer. When the display timer fires a rendering update is done and a shared memory for the frame is allocated. The updated rectangles are painted directly into the shared memory buffer and the Update message is sent to the UI process with the list of updated rectangles and the shared memory file descriptor. The accelerated mode is a lot more complex, the web contents are split into layers depending on the properties of each element. So, in addition to the render tree, in accelerated mode we also have a layer tree. Every layer handles its contents separately producing a GL texture. The layer tree is then composed to produce the final frame that is shared with the UI process.

The layer tree¶

Every RenderView has a RenderLayerCompositor to create and handle the layers for the RenderView contents. When entering in accelerated compositing mode the RenderLayerCompositor creates the root layer. GraphicsLayer is the base class used to represent the layers, and requires a derived class for every platform implementation (CoordinatedGraphics, CoreAnimation, etc.). GraphicsLayer::create() receives a GraphicsLayerFactory to create the derived class for the current platform, and a GraphicsLayerClient that is the RenderLayerCompositor for the layers it creates. GraphicsLayerFactory is an abstract class with a single pure virtual method createGraphicsLayer() that derived classes must implement to create the platform specific GraphicsLayer. The RenderLayerCompositor gets the GraphicsLayerFactory from its page ChromeClient, that in the WebKit implementation uses the DrawingArea to get it. LayerTreeHost is the class implementing the GraphicsLayerFactory interface for ports using coordinated graphics. A LayerTreeHost is created by the DrawingAreaCoordinatedGraphics when entering accelerated compositing mode. LayerTreeHost::createGraphicsLayer() simply creates a new GraphicsLayerCoordinated that is the coordinated graphics implementation of GraphicsLayer. A GraphicsLayer has a set of properties like size, position, opacity, etc. and contents that can change at any time due to an input event, a running animation, a JavaScript call, etc. A change in a GraphicsLayer usually requires a repaint, so whenever a property changes the property is updated and a repaint is scheduled if needed. RenderingUpdateScheduler class triggers a rendering update using display link to schedule repaint on display refresh that ends up calling DrawingAreaCoordinatedGraphics::triggerRenderingUpdate. In accelerated mode a layer flush is scheduled on the LayerTreeHost, that uses a timer to schedule it. LayerTreeHost::flushLayers() calls WebPage::updateRendering() that runs all the steps to render a page, and then WebPage::finalizeRenderingUpdate() that runs the actual layer flush and applies the scrolling tree layers position if async scrolling is enabled. Page::finalizeRenderingUpdate() iterates all its root frames calling Page::finalizeRenderingUpdateForRootFrame() that calls LocalFrameView::flushCompositingStateForThisFrame() that calls RenderLayerCompositor::flushPendingLayerChanges() on its RenderView compositor. The RenderLayerCompositor calls GraphicsLayer::flushCompositingState on the root layer and finally GraphicsLayerCoordinated::flushCompositingState() iterates the layer tree recursively to commit all pending changes in each layer. Coordinated graphics composes the layer tree in a secondary thread, so GraphicsLayer objects can't be used directly by the compositor, since they are not thread safe. An intermediate thread safe object CoordinatedPlatformLayer is used to accumulate the GraphicsLayer state until it's used by the compositor thread to produce the final frame. The CoordinatedPlatformLayer object has an owner that is the GraphicsLayer, a client that is the LayerTreeHost and a target layer that is a TextureMapperLayer. On layer flush, the properties that changed in GraphicsLasyer are set to its CoordinatedPlatformLayer. Whenever a property changes in CoordinatedPlatformLayer a composition is requested. CoordinatedPlatformLayer is also responsible for producing the layer contents as a texture. When the compositor executes a request, it iterates the CoordinatedPlatformLayer list running another flush that calls CoordinatedPlatformLayer::flushCompositingState() to set the current state into the final TextureMapperLayer that will be used by TextureMapper to compose the frame.

But not only existing layer properties can change at any time, the layer tree itself can change too due to added or removed layers. This means that the layer tree at the time a composition request might have changed once the composition request is executed in the compositor thread. To make sure that the compositor always uses the current tree LayerTreeHost uses CoordinatedSceneState. It's a thread safe object that keeps track of the CoordinatedPlatformLayers managed by the LayerTreeHost. The scene state object is notified whenever a layer is attached to or detached from the LayerTreeHost. CoordinatedSceneState keeps its own list of CoordinatedPlatformLayers with a flag to indicate that the tree has changed. The compositor is created with a reference to the CoordinatedSceneState from its LayerTreeHost. The compositor flushes the CoordinatedPlatformLayers by getting a copy of the current list of layers from CoordinatedSceneState.

BackingStore¶

A layer that needs to paint its contents uses a backing store object. During the layer flush, once all properties have been committed (set on the CoordinatedPlatformLayer), CoordinatedPlatformLayer::updateContents() is called. It checks if a backing store is needed, the condition is that the layer has the draws content flag enabled, it's visible and size is not empty. If it's needed then a CoordinatedBackingStoreProxy is created if it wasn't already created and the layer is marked as needing tile updates. Once the root GraphicsLayer has committed all properties of all its children recursively it calls GraphicsLayerCoordinated::updateBackingStoresIfNeeded(). This method iterates the tree again to call updateBackingStore() on all CoordinatedPlatformLayers. CoordinatedPlatformLayers::updateBackingStore() checks if there's a backing store that needs an update because there's a dirty region, tiles needs an update or there are more tiles pending to be created. When a layer needs to be repainted for whatever reason setNeedsDisplay() or setNeedsDisplayInRect() are called on GraphicsLayer generating a dirty region to be repainted. A GraphicsLayer also needs to update the tiles when a new backing store is created or when the layer transformation changes. When a repaint is needed CoordinatedBackingStoreProxy::updateIfNeeded() is called. The backing store splits the layer into tiles of 512x512. During the update it checks if tiles need to be created or removed. It also checks if tiles need to be updated according to the passed dirty region. The tiles are iterated checking if they intersect with the dirty region. Every tile produces a single dirty rectangle (the union of all rectangles inside the tile that intersect with the dirty region), for which a paint task is scheduled calling CoordinatedPlatformLayer::paint(). The SkiaPaintingEngine then sends the painting task to the main, CPU or GPU thread depending on the configuration. The result is a CoordinatedTileBuffer where the tile dirty rectangle is rendered into. At the end of CoordinatedBackingStoreProxy::updateIfNeeded() we have a list of tiles to be added, removed and updated that are merged with a previous pending update if any. The CoordinatedBackingStoreProxy also pre-renders more tiles to ensure they are ready in case of scrolling, but those are follow up requests. So, the result of CoordinatedBackingStoreProxy::updateIfNeeded() is an OptionSet with flags to indicate whether more tiles need to be painted, whether any tile has been scheduled to be painted and whether there's any tile change (added, removed or updated). If more tiles need to be painted the GraphicsLayer notifies its client (the RenderLayerCompositor) calling notifySubsequentFlushRequired() to ensure a new layer flush is scheduled right after this one (note that this subsequent flush will not cause a composition if nothing changed). On the compositor side the CoordinatedBackingStoreProxy pending request is consumed by CoordinatedPlatformLayer in CoordinatedPlatformLayer::flushCompositingState(). A CoordinatedBackingStore is created if needed and the list of tiles to create, remove or update are set to the backing store and then CoordinatedBackingStore::processPendingUpdates() is called. Tiles are iterated calling CoordinatedBackingStoreTile::processPendingUpdates() that iterates the list of pending updates to get the buffer of the tile dirty region. Every CoordinatedBackingStoreTile has its own texture to cover the whole tile rectangle, but the updates can contain a dirty region that is smaller than the tile size. Depending on the dirty rectangle and the type of the dirty region buffer the possible actions could be:

If the buffer is not accelerated: contents need to be submitted to the tile texture. BitmapTexture::updateContents() is called to upload the buffer to the texture. In this case there's no difference between having a dirty region covering the whole tile area or not.
If the buffer is accelerated and dirty region is smaller than tile area: the dirty region texture needs to be copied into the tile texture. BitmapTexture::copyFromExternalTexture() is called to perform a GPU to GPU copy.
If the buffer is accelerated and dirty region is the same as the tile area: in this case we can just use the dirty region texture as the tile texture without any copies. BitmapTexture::swapTexture() is called in this case.

Later, when the texture mapper paints the layer CoordinatedBackingStore::paintToTextureMapper() is called. This iterates the tiles calling TextureMapper::drawTexture() with each tile texture.

For layers that don't draw their contents, the contents are provided in different ways using CoordinatedPlatformLayerBuffer.

Accelerated images¶

Images are usually painted into layers as part of the backing store painting. But in some cases, an image has its own layer, and a different backing store is used, the CoordinatedImageBackingStore. When the contents of a layer is a single image, GraphicsLayer::setContentsToImage() is called to set the image. If the given image is different to the current one, it's saved and the layer noted as changed. During the layer flush CoordinatedPlatformLayer::setContentsImage() is called, which uses a double-buffer like CoordinatedImageBackingStore. If there's no current image backing store or it contains a different image, the client (LayerTreeHost) is asked for a new CoordinatedImageBackingStore for the given image. LayerTreeHost handles the image backing stores because it keeps a cache of images, so that if there are multiple layers with the same image as contents, the same CoordinatedImageBackingStore is used for all the layers. CoordinatedImageBackingStore is just a thread safe ref counted wrapper around a CoordinatedPlatformLayerBufferNativeImage that contains the platform image. The CoordinatedPlatformLayerBufferNativeImage contains another CoordinatedPlatformLayerBuffer to store the native image texture. If the native image is accelerated the buffer is created on construction in the main thread as a CoordinatedPlatformLayerBufferRGB that receives the image texture and a fence. If the native image is not accelerated the buffer is created on composition in the compositor thread when it's requested to be painted into the texture mapper. The image contents are then uploaded to a newly created texture (or reused from texture mapper pool) that is passed to CoordinatedPlatformLayerBufferRGB::create(). For subsequent paint requests the internal buffer already exists and it's reused.

Delegated content¶

Delegated content is used for WebGL, accelerated 2D canvas and Offscreen canvas. When CanvasRenderingContext::setContentsToLayer() is called to set the layer, GraphicsLayer::setContentsDisplayDelegate() is called for the given layer using the GraphicsLayerContentsDisplayDelegate returned by virtual method GraphicsLayerContentsDisplayDelegate::layerContentsDisplayDelegate(). The base class GraphicsLayerContentsDisplayDelegate contains two pure virtual methods: setDisplayBuffer(), that receives a CoordinatedPlatformLayerBuffer, and display() that receives the CoordinatedPlatformLayer that should use the buffer as contents. GraphicsLayer::setContentsDisplayDelegate() just sets the contents delegate and marks the layer as changed (both contents buffer and contents buffer needs display). Whenever a repaint is required for a layer using delegated content, GraphicsLayerContentsDisplayDelegate::setDisplayBuffer() is called to set the buffer and GraphicsLayer::setContentsNeedsDisplay() is called marking the layer as changed. On layer flush if contents need display GraphicsLayerContentsDisplayDelegate::display() is called. The implementation of GraphicsLayerContentsDisplayDelegate::display() then should call CoordinatedPlatformLayer::setContentsBuffer() to set the CoordinatedPlatformLayerBuffer for the current delegated content. CoordinatedPlatformLayerBuffer also uses a double-buffer for its contents buffers, so the given buffer is set as pending. On composition flush if layer contents buffer changed, the pending is set as committed and the committed one is set in the TextureMapperLayer calling TextureMapperLayer::setContentsLayer(). GraphicsLayerContentsDisplayDelegateCoordinated is the coordinated graphics implementation of GraphicsLayerContentsDisplayDelegate, that keeps a CoordinatedPlatformLayerBuffer set in setDisplayBuffer() and passed to the CoordinatedPlatformLayerBuffer in display(). The main difference of the GraphicsLayerContentsDisplayDelegateCoordinated users is the type of CoordinatedPlatformLayerBuffer that they pass to setDisplayBuffer().

Accelerated 2D canvas: in prepareForDisplay() the canvas contents are extracted as an accelerated image that is used to create a CoordinatedPlatformLayerBufferNativeImage.
WebGL without GBM: in prepareForDisplay() a CoordinatedPlatformLayerBufferRGB is created for the current display texture.
WebGL with GBM: in prepareForDisplay() a CoordinatedPlatformLayerBufferDMABuf is created for the current display buffer.
Offscreen canvas: this case is a bit different because it uses a GraphicsLayerAsyncContentsDisplayDelegate to copy the canvas from different threads, which contains a pure virtual method tryCopyToLayer() that receives an ImageBuffer. GraphicsLayerAsyncContentsDisplayDelegateCoordinated is the coordinated graphics implementation of GraphicsLayerAsyncContentsDisplayDelegate. It creates a non-async GraphicsLayerContentsDisplayDelegate that is set on the given layer on construction. In tryCopyToLayer() the given ImageBuffer is cloned and its native image is used to create a CoordinatedPlatformLayerBufferNativeImage that is set to the non-async delegate calling setDisplayBuffer().

Media¶

In the case of media, the video frames are provided by the media player at its own pace and updates are not handled by layer flush, but passed directly to the compositor. The class used to handle video frames is CoordinatedPlatformLayerBufferProxy. The media player creates a CoordinatedPlatformLayerBufferProxy on construction that is returned as its platform layer. GraphicsLayer::setContentsToPlatformLayer() is called to set the CoordinatedPlatformLayerBufferProxy on the graphics layer. GraphicsLayer::setContentsToPlatformLayer() calls CoordinatedPlatformLayerBufferProxy::setTargetLayer() passing its CoordinatedPlatformLayer to associate the proxy with the platform layer. The first time the proxy is set, the layer is marked as changed as usual and if at the time of layer flush there was a pending buffer, it's set to the platform layer calling CoordinatedPlatformLayer::setContentsBuffer() the same way it's done for delegated content. The next frames are handled directly by the proxy and sent to the compositor without a layer flush. Whenever a new frame is available (usually from a GStreamer thread) the media player creates a CoordinatedPlatformLayerBufferVideo for the current frame and calls CoordinatedPlatformLayerBufferProxy::setDisplayBuffer(). CoordinatedPlatformLayerBufferProxy::setDisplayBuffer() saves the buffer as pending if the proxy hasn't been attached to a platform layer or calls CoordinatedPlatformLayer::setContentsBuffer() on the attached platform layer and asks the layer to request a composition to its client (LayerTreeHost). In CoordinatedPlatformLayer the media buffers are handled exactly the same way as the delegated content buffers. CoordinatedPlatformLayerBufferVideo is a class that contains another CoordinatedPlatformLayerBuffer that depends on the type of video frame and whether the frame contains a DMA-BUF buffer, GL texture or main memory.

If the frame contains a DMA-BUF buffer, a CoordinatedPlatformLayerBufferDMABuf is created for the imported DMA-BUF.
If the frame contains a GL texture:
A CoordinatedPlatformLayerBufferExternalOES is created if the frame contains an external OES texture.
A CoordinatedPlatformLayerBufferRGB is created if the frame contains a single plane texture.
A CoordinatedPlatformLayerBufferYUV is created if the frame contains a supported YUV multiplane texture.
If the frame contains main memory the frame is mapped on construction and a CoordinatedPlatformLayerBufferRGB is created for a newly created texture (or taken from the texture mapper pool), for which the frame contents are uploaded, from the compositor thread when the buffer is requested to be painted into the texture mapper.

The compositor¶

The compositor runs in a separate thread in the web process. It's created by the LayerTreeHost on construction. When the compositor is created it performs the following actions:

Creates an AcceleratedSurface to render the composited frames that will be shared with the UI process.
Takes a reference to the CoordinatedSceneState of the LayerTreeHost to keep track of the layers.
Creates a CompositingRunLoop that will drive the compositor tasks.
Notifies the surface that the compositing run loop has been created, which is implemented by AcceleratedSurfaceDMABuf to initialize the IPC message receiver to receive messages from UI process in the compositor thread.
Synchronously creates the GL context in the compositor thread by scheduling a sync task.

After that the compositor is idle waiting for requests. It can receive requests in different ways:

ThreadedCompositor::requestComposition(): This is called by LayerTreeHost on layer flush when a composition is required. It returns a request ID that the LayerTreeHost uses to know when this particular request has been completed.
ThreadedCompositor::scheduleUpdate(): This is called by LayerTreeHost when a new composition is required for async scrolling or media player, or by the compositor itself when there are active animations. In this case there's no request identifier so the last one set by requestComposition() is used.

When a new composition is requested either by requestComposition() or scheduleUpdate() an update is scheduled in the CompositingRunLoop. The CompositingRunLoop only allows one composition at a time per frame so the following can happen:

The loop is idle: state is set as scheduled and a 0 timer is started to run the update function (renderLayerTree).
The loop is scheduled: does nothing, update function will be called when the timer fires.
The loop is "in progress": the state is marked as having a pending update to be run once the loop is idle again.

When the update timer fires, the state is set to "in progress" and the update function is called (renderLayerTree). It will stay in "in progress" state until UI notifies that the current frame has been presented on screen (the FrameDone message). Then CompositingRunLoop::updateCompleted() is called and the following can happen in this case:

The loop is idle or scheduled: does nothing.
The loop is "in progress": if the state has a pending update, the state is changed as scheduled and the update timer is started, otherwise the state is set to idle.

The compositor uses texture mapper to render all layers into the final frame. This happens in ThreadedCompositor::renderLayerTree() that is called by the CompositingRunLoop when the update timer fires. The following steps are run:

Scene attributes (size and device scale factor) are used to create a TransformationMatrix and resize the surface if needed.
The surface is notified that a new frame is about to be rendered. The surface prepares the next target allocating new buffers if needed and preparing the framebuffer.
LayerTreeHost is also notified scheduling a main thread call.
If the surface was resized, glViewport() is called with the new size.
Surface is asked to clear if needed to call glClear() with transparent color if the surface is not opaque.
Scene is updated: flushCompositingState() is called for all the currently active layers in CoordinatedSceneState.
Layers are now painted into the texture mapper.
If there are running animations another composition is requested.
The composition response identifier is set to the current request identifier and LayerTreeHost is notified calling didComposite() from a main thread timer passing the composition response identifier.
The surface is notified that a frame has been rendered. In the case of AcceleratedSurfaceDMABuf the Frame message is sent to the UI process.
LayerTreeHost is also notified scheduling a main thread call.

The LayerTreeHost ensures layer flushes can't be done while there's an ongoing composition by setting a variable (m_isWaitingForRenderer) before calling ThreadedCompositor::requestComposition() and unsetting it when the composition is done in LayerTreeHost::didComposite() if the given response identifier matches the current request identifier. If a layer flush is requested while m_isWaitingForRenderer is enabled, another variable is used to indicate that a layer flush was requested but couldn't be done. When current composition finishes, if a layer flush was scheduled while m_isWaitingForRenderer was true, the pending layer flush is performed immediately.

Async scrolling¶

Async scrolling handles wheel events in a secondary thread in the web process to try to ensure we respond quickly to scrolling requests with visual feedback. The idea is that the scrolling thread can schedule a composition without going through the main thread if rendering layers is taking too long since the scrolling was requested. Wheel events are not sent from WebPageProxy to WebPage like other events, they are sent from WebPageProxy to EventDispatcher to be processed by the web process in a separate thread. The EventDispatcher is a singleton running in the web process that uses a work queue to handle IPC messages off the main thread. When async scrolling is enabled DrawingAreaCoordinatedGraphics registers the scrolling tree in the event dispatcher by calling EventDispatcher::addScrollingTreeForPage(). EventDispatcher::wheelEvent() handles the IPC message from the work queue thread. If there isn't a scrolling tree registered for the given page identifier the event is scheduled to be handled in the main thread. Otherwise the event is dispatched by the scrolling thread (from the EventDispatcher thread). When executed in the scrolling thread ScrollingTree::handleWheelEvent() is called, which determines the scrolling tree node that should handle the event for the given position and calls ScrollingTree::handleWheelEventWithNode(). The scrolling tree is iterated from the given node calling ScrollingTreeNode::handleWheelEvent() until the event is handled by a node. handleWheelEvent() is a virtual method that just returns "unhandled" by default, so it needs to be implemented by derived classes. Events are handled by a delegate, so derived classes usually just forward the event handling to their delegate. ScrollingTreeScrollingNodeDelegateCoordinated is the coordinated graphics implementation of ThreadedScrollingTreeScrollingNodeDelegate used by the coordinated graphics scrolling nodes to handle scrolling events. The delegate uses a ScrollingEffectsController to process the events, so ScrollingEffectsController::handleWheelEvent() is what actually processes the event from the scrolling thread (the same way it's done in the main thread by ScrollAnimator class). The scroll deltas are calculated and then an immediate or animated scrolling is started by asking the ScrollingEffectsController client (in this case is the delegate). The delegate calls scrollBy() on its scrolling tree node passing the given scroll deltas. The node calculates its new position for the scrolling deltas and the scrolling tree is notified that the node has been scrolled. ThreadedScrollingTree::scrollingTreeNodeDidScroll() generates a ScrollUpdate for the node and notifies the scrolling coordinator from the main thread, which schedules a rendering update. When RenderingUpdateScheduler schedules the update the scrolling coordinator is also notified. After iterating the nodes EventDispatcher notifies the UI process by sending the message DidReceiveEvent to the WebPageProxy indicating whether it was handled or not. At this point the scrolling tree nodes have updated their position for the scrolling event, and the main thread has scheduled a rendering update. When the layer flush is triggered the following happens:

From Page::updateRendering(), which is called right before the layer properties are committed, willStartRenderingUpdate() is called on the scrolling coordinator, which notifies its tree and calls synchronizeStateFromScrollingTree() to update.
ThreadedScrollingTree::willStartRenderingUpdate(): sends a sync task to the scrolling thread to block until current rendering update is completed or until a timeout of at most half a frame duration fires. The tree state is then set as InRenderingUpdate. If the scrolling thread is unblocked because of the timeout, it means the rendering update hasn't completed yet and the tree state is set as Desynchronized. If layer positions can be updated from the scrolling thread applyLayerPositions() is called in a follow up run loop iteration inside the scrolling thread.
AsyncScrollingCoordinator::synchronizeStateFromScrollingTree(): First calls applyPendingScrollUpdates() that iterates all pending updates calling applyScrollPositionUpdate(). For position updates updateScrollPositionAfterAsyncScroll() is called which calls reconcileScrollingState() if the given node id is the frame view one. This ends up calling GraphicsLayer::syncPosition() for the frame view scroll container layer and all other scroll related layers if present. GraphicsLayer::syncPosition() is like GraphicsLayer::setPosition(), but the new value is not set on the CoordinatedPlatformLayer when properties are committed during the layer flush, since the platform layer is expected to be updated directly by the scrolling tree node.
From Page::finalizeRenderingUpdate(), right after all layer properties have been committed, applyScrollingTreeLayerPositions() and didCompleteRenderingUpdate() are called for the page scrolling coordinator.
AsyncScrollingCoordinator::applyScrollingTreeLayerPositions() just calls ScrollingTree::applyLayerPositions() which recursively iterates the scrolling tree to call applyLayerPositions() for every node. All the nodes end up setting their updated position to their associated CoordinatedPlatformLayer that will request a composition if the position changed. At this point the GraphicsLayer and its CoordinatedPlatformLayer are in sync again.
ScrollingTreeCoordinated::didCompleteRenderingUpdate(): layers are now committed, but rendering is asynchronous with coordinated graphics, so at this point the layers are being painted if needed and a composition was requested if any property changed during the layer flush. So, this checks if a composition is ongoing or has been requested and returns early in that case to wait for the composition to complete. Otherwise renderingUpdateComplete() is called to unblock the scrolling thread if it's still blocked waiting for the rendering.

If the layer flush triggered a composition, LayerTreeHost notifies the page when the composition is done by calling Page::didCompleteRenderingUpdateDisplay() which calls didCompletePlatformRenderingUpdate() for the scrolling coordinator. ScrollingTreeCoordinated::didCompletePlatformRenderingUpdate() just calls renderingUpdateComplete() to unblock the scrolling thread if it's still blocked waiting for the rendering.

ThreadedScrollingTree is also synchronized with the display link. EventDispatcher is also notified from the UI process when display has been refreshed, and notifies all the registered scrolling trees calling displayDidRefresh(). As with the events, this is done from work queue thread to scrolling thread without passing through the main thread. On display refresh the scrolling thread does the following:

Calls serviceScrollAnimations() which calls serviceScrollAnimation() on every scrolling node with an active scroll animation. This calls the animation callback of the ScrollingEffectsController.
If state is not idle and layers can be updated from the scrolling thread, applyLayerPositions() is called. This ends up updating the CoordinatedPlatformLayer position like during the rendering update but now from the scrolling thread instead of the main thread. In this case a composition is requested directly from the scrolling thread after positions have been applied.
If state is still idle and a rendering update was scheduled then state is set as WaitingForRenderingUpdate and a 1ms timer is started. The timer is stopped right before the scrolling thread is blocked to wait for the rendering update. If the timer fires before the next rendering update starts, applyLayerPositions() is called, if layers can be updated from the scrolling thread, and state is set to Desynchronized.