안드로이드 미디어 프레임워크(SurfaceFlinger) 학습 2

안드로이드 미디어 프레임워크(SurfaceFlinger) 학습 1 (tistory.com)

안드로이드 미디어 프레임워크(SurfaceFlinger) 학습 1

지난 번의 분석은 queueBuffer를 해주는 여러 가지 중 하나를 확인을 해봤었습니다
 
이번에 분석을 해볼 사항은, SurfaceFlinger에 관해서 알아보려고 합니다
SurfaceFlinger는 layer단위로 관리를 하고 있습니다 사용되는 layer의 class diagram은 아래와 같습니다
 

SurfaceFlinger에서의 layer

 
그렇다면 이 layer가 create되는 건 어떻게 되는 걸까요?
 

CreateLayer

---

// 1.SurfaceFlinger::addClientLayer
    // Create a transaction includes the initial parent and producer.
    Vector<ComposerState> states;
    Vector<DisplayState> displays;

    ComposerState composerState;
    composerState.state.what = layer_state_t::eLayerCreated;
    composerState.state.surface = handle;
    states.add(composerState);
    
    ...
    
    return setTransactionState(FrameTimelineInfo{}, states, displays, 0 /* flags */, nullptr,
                               InputWindowCommands{}, -1 /* desiredPresentTime */,
                               true /* isAutoTimestamp */, {}, false /* hasListenerCallbacks */, {},
                               0 /* Undefined transactionId */);
                               
// 2. SurfaceFlinger::setTransactionState
     TransactionState state{frameTimelineInfo,  states,
                           displays,           flags,
                           applyToken,         inputWindowCommands,
                           desiredPresentTime, isAutoTimestamp,
                           uncacheBuffer,      postTime,
                           permissions,        hasListenerCallbacks,
                           listenerCallbacks,  originPid,
                           originUid,          transactionId};

    // Check for incoming buffer updates and increment the pending buffer count.
    state.traverseStatesWithBuffers([&](const layer_state_t& state) {
        mBufferCountTracker.increment(state.surface->localBinder());
    });
    queueTransaction(state);
    
 // 3. SurfaceFlinger::queueTransaction
        mTransactionQueue.emplace(state);

1. addClientLayer를 실행할 경우 ComposerState에 surface와 layerCreated state를 채운 후 States에 추가하여 넘겨주고
2. setTransactionState 실행 시 queueTransaction을 실행하여, state를 넘겨 줍니다
3. queueTransaction에서는 넘겨받은 state를 mTransactionQueue에 채웁니다
 

// SurfaceFlinger::flushTransactionQueue
			// Collect transactions from current transaction queue or queue to pending transactions.
            // Case 1: push to pending when transactionIsReadyToBeApplied is false.
            // Case 2: push to pending when there exist a pending queue.
            // Case 3: others are ready to apply.
            while (!mTransactionQueue.empty()) {
                auto& transaction = mTransactionQueue.front();
                bool pendingTransactions = mPendingTransactionQueues.find(transaction.applyToken) !=
                        mPendingTransactionQueues.end();
                if (pendingTransactions ||
                    !transactionIsReadyToBeApplied(transaction.frameTimelineInfo,
                                                   transaction.isAutoTimestamp,
                                                   transaction.desiredPresentTime,
                                                   transaction.originUid, transaction.states,
                                                   bufferLayersReadyToPresent)) {
                    mPendingTransactionQueues[transaction.applyToken].push(std::move(transaction));
                } else {
                    transaction.traverseStatesWithBuffers([&](const layer_state_t& state) {
                        bufferLayersReadyToPresent.insert(state.surface);
                    });
                    transactions.emplace_back(std::move(transaction));
                }
                mTransactionQueue.pop();
                ATRACE_INT("TransactionQueue", mTransactionQueue.size());
            }
        }

        // Now apply all transactions.
        for (const auto& transaction : transactions) {
            applyTransactionState(transaction.frameTimelineInfo, transaction.states,
                                  transaction.displays, transaction.flags,
                                  transaction.inputWindowCommands, transaction.desiredPresentTime,
                                  transaction.isAutoTimestamp, transaction.buffer,
                                  transaction.postTime, transaction.permissions,
                                  transaction.hasListenerCallbacks, transaction.listenerCallbacks,
                                  transaction.originPid, transaction.originUid, transaction.id);
            if (transaction.transactionCommittedSignal) {
                mTransactionCommittedSignals.emplace_back(
                        std::move(transaction.transactionCommittedSignal));
            }
        }
    }

flushTransactionQueues가 불릴 때, mTransactionQueue에 넣어놨던 값들을 보며 
적용 가능하거나 혹은, 이전에 pending된 transaction이 아니면 applyTransactionState를 실행 해줍니다
 
applyTransactionState가 실행되면

uint32_t SurfaceFlinger::setClientStateLocked(
        const FrameTimelineInfo& frameTimelineInfo, const ComposerState& composerState,
        int64_t desiredPresentTime, bool isAutoTimestamp, int64_t postTime, uint32_t permissions,
        std::unordered_set<ListenerCallbacks, ListenerCallbacksHash>& outListenerCallbacks) {
    const layer_state_t& s = composerState.state;
    
    ...
    
    sp<Layer> layer = nullptr;
    if (s.surface) {
        if (what & layer_state_t::eLayerCreated) {
            layer = handleLayerCreatedLocked(s.surface);
            if (layer) {
                flags |= eTransactionNeeded | eTraversalNeeded;
                mLayersAdded = true;
            }
        } else {
            layer = fromHandle(s.surface).promote();
        }
    }

그렇게 되면 what에 대해서 필요한 상황에 대한 state를 set해줍니다
 

SurfaceFlinger 역할

SurfaceFlinger에서는 mDrawingState 라는 자료 구조안에 layer를 z 값으로 정렬 해놓은 layersSortedByZ를 가지고 layer 값들을 가지고 있습니다
이후, hw 단에서 vsync와 같은 signal이 오면, layer단의 정보를 넘기며 화면에 compose 해달라는 요청을 내립니다

// 1. SurfaceFlinger::onMessageRefresh
    compositionengine::CompositionRefreshArgs refreshArgs;
    const auto& displays = ON_MAIN_THREAD(mDisplays);
    refreshArgs.outputs.reserve(displays.size());
    for (const auto& [_, display] : displays) {
        refreshArgs.outputs.push_back(display->getCompositionDisplay());
    }
    mDrawingState.traverseInZOrder([&refreshArgs](Layer* layer) {
        if (auto layerFE = layer->getCompositionEngineLayerFE())
            refreshArgs.layers.push_back(layerFE);
    });
    refreshArgs.layersWithQueuedFrames.reserve(mLayersWithQueuedFrames.size());
    for (auto layer : mLayersWithQueuedFrames) {
        if (auto layerFE = layer->getCompositionEngineLayerFE())
            refreshArgs.layersWithQueuedFrames.push_back(layerFE);
    }
    
    ...
    
    mCompositionEngine->present(refreshArgs);
    
// 2. CompositionEngine::present

    updateLayerStateFromFE(args);

    for (const auto& output : args.outputs) {
        output->present(args);
    }

1) onMessageRefresh가 불리면 드디어 화면에 보여주기 위한 것들을 넘기며 present를 호출합니다
2) CompositionEngine은 Output에 present 요청을 합니다
 
실제로 compose 하는 부분은 hwComposer로서 vendor chipset 부분의 구현 부분에 따라 다를 수 있는 것으로 보입니다
일단은 aosp 부분의 코드를 계속 따라가 보려고 합니다
 

surfaceFlinger에서 사용하는 nativeWindow

SurfaceFlinger에서는 display가 추가 되면 DisplaySurface를 만듭니다(FramebufferSurface)
RenderSurfaceCreationArgs의 displaysurface와, NativeWindow에 각각 넣어서 RenderSurface를 만들 때 인자로 같이 넣어줍니다
 
고로 실질적으로 Output에서 작업하는 surface는 RenderSurface입니다 

// Output::present
void Output::present(const compositionengine::CompositionRefreshArgs& refreshArgs) {
    ATRACE_CALL();
    ALOGV(__FUNCTION__);

    updateColorProfile(refreshArgs);
    updateCompositionState(refreshArgs);
    planComposition();
    writeCompositionState(refreshArgs);
    setColorTransform(refreshArgs);
    beginFrame();
    prepareFrame();
    devOptRepaintFlash(refreshArgs);
    finishFrame(refreshArgs);
    postFramebuffer();
    renderCachedSets(refreshArgs);
}