There is an increasing amount of empirical work investigating the sense of agency, i.e. the registration that we are the initiators of our own actions. Many studies try to relate the sense of agency to an internal feed-forward mechanism, called the "comparator model". In this paper, we draw a sharp distinction between a non-conceptual level of feeling of agency and a conceptual level of judgement of agency. By analyzing recent empirical studies, we show that the comparator model is not able to explain either. Rather, we argue for a two-step account: a multifactorial weighting process of different agency indicators accounts for the feeling of agency, which is, in a second step, further processed by conceptual modules to form an attribution judgement. This new framework is then applied to disruptions of agency in schizophrenia, for which the comparator model also fails. Two further extensions are discussed: We show that the comparator model can neither be extended to account for the sense of ownership (which also has to be differentiated into a feeling and a judgement of ownership) nor for the sense of agency for thoughts. Our framework, however, is able to provide a unified account for the sense of agency for both actions and thoughts.