Aspect-based financial sentiment analysis (ABFSA) is a fine-grained task that can enrich and sharpen financial analysis by identifying sentiments towards specific entities (e.g., company, stock). As the application of ABSA in professional language, ABFSA is a challenging task requiring extensive domain knowledge while remaining understudied. Numeral understanding is crucial for financial text analysis, but existing NLP models for ABFSA lack such ability by mainly treating numerals as plain text. In addition, most studies on knowledge incorporation disregard necessary domain-specific connotations or suffer from the low coverage issue. In this paper, we propose a novel numeral-oriented network with a multi-source affective knowledge refinement strategy (NumAKEN) for ABFSA. NumAKEN utilizes a numeral encoding method based on DigitCNN to capture critical numeric concepts such as magnitude and category. A multi-source affective knowledge fusion strategy is designed for hybrid lexicon construction and incorporation, which can guide the model to capture significant sentiment clues as well as alleviate conflicting and coverage issues. Extensive experiments on two datasets illustrate that our NumAKEN model outperforms all state-of-the-art methods and verify the effectiveness of our model.