Character Animation with Direct3D- P13


lượt xem

Character Animation with Direct3D- P13

Mô tả tài liệu
  Download Vui lòng tải xuống để xem tài liệu đầy đủ


Chủ đề:

Nội dung Text: Character Animation with Direct3D- P13

  1. 226 Character Animation with Direct3D fairly educated guess as to which phoneme is being spoken. Figure 10.3 shows the waveform from the previous speech sample together with the spectrograph of the same sample (a spectrograph shows the frequency and amplitude of a signal). FIGURE 10.3 Waveform and spectrograph of a voice sample. As you can see in Figure 10.3, distinct patterns can be seen in the spectrograph as the phonemes are spoken. As a side note, speech-to-text applications take this analysis one step further and use Hidden Markov Models (HMM) to figure out which exact word is being spoken. Luckily we don’t need to dive that deep in order to create reasonable lip-syncing. If you are interested in analyzing speech and making your own phoneme extractor, you’ll need to run the speech data through a Fourier Transform. This will give you the data in the frequency domain, which in turn will help you build the spectrogram and help you classify the phonemes. Check out for a Fast-Fourier- Transform library in C. Analyzing speech and extracting phonemes is a rather CPU-intense process and is therefore pre-processed in all major game engines. However, some games in the past have used a real-time lip-syncing system based simply on the current am- plitude of the speech [Simpson04]. With this approach the voice lines are evaluated just a little ahead of the playback position to determine which mouth shape to use. In the coming sections I will look at a similar system and get you started on ana- lyzing raw speech data. Please purchase PDF Split-Merge on to remove this watermark.
  2. Chapter 10 Making Characters Talk 227 S OUND D ATA Before you can start to analyze voice data, I’ll need to go off on a tangent and cover how to actually load some raw sound data. No matter which type of sound format you will actually use, once the sound data has been decompressed and decoded, the raw sound data will be the same across all sound formats. In this chapter I’ll just use standard uncompressed WAVE files for storing sound data. However, for projects requiring large amounts of voice lines, using uncompressed sound is of course out of the question. Two open-source compression schemes available for free are OGG and SPEEX, which you can find online: OGG is aimed mainly at music compression and streaming, but it is easy enough to get up and running. SPEEX, on the other hand, focuses only on speech com- pression. THE WAVE FORMAT There are several good tutorials on the Web explaining how to load and interpret WAVE (.wav) files, so I won’t dig too deep into it here. The WAVE format builds on the Resource Interchange File Format (RIFF). RIFF files store data in chunks, where the start of each chunk is marked with a 4-byte ID describing what type of chunk it is, as well as 4 bytes containing the size of the chunk (a long). The WAVE file con- tains all information about the sound—number of channels, sampling rate, number of bits per sample, and much more. Figure 10.4 shows how a WAVE file is organized. There are many other different types of chunks that can be stored in a WAVE file. Only the Format and Data chunks are mandatory. Table 10.2 shows the differ- ent fields of the Format chunk and their possible values. ease purchase PDF Split-Merge on to remove this watermark.
  3. 228 Character Animation with Direct3D FIGURE 10.4 WAVE file format. TABLE 10.2 THE WAVE FORMAT CHUNK Field Type Description Audio Format Short Type of audio data. A value of 1 indicates PCM data; other values mean that there’s some form of compression. Num Channels Short Number of channels, 1 = mono, 2 = stereo Sample Rate Long Number of samples per second. For example, CD quality uses 44,100 samples per second (Hz). Byte Rate Long Number of bytes used per second. Block Align Short Number of bytes per sample (including multiple channels). Bits/Sample Short 8 = 8 bits per sample, 16 = 16 bits per sample. Please purchase PDF Split-Merge on to remove this watermark.
  4. Chapter 10 Making Characters Talk 229 For a full description of the WAVE file format and the different chunks avail- able, check out I’ll assume that the data stored in the “data” chunk is uncompressed sound data in the Pulse-Code Modulation (PCM) format. This basically means that the data is stored as a long array of values where each value is the amplitude of the sound at a specific point in time. The quickest and dirtiest way to access the data is to simply open a stream from the sound file and start reading from byte 44 (where the data field starts). Although this will work if you know the sound specifications, it isn’t really recommended. The WaveFile class I’ll present here will do minimal error checking before reading and storing the actual sound data: class WaveFile { public: WaveFile(); ~WaveFile(); void Load(string filename); short GetMaxAmplitude(); short GetAverageAmplitude(float startTime, float endTime); float GetLength(); public: long m_numSamples; long m_sampleRate; short m_bitsPerSample; short m_numChannels; short *m_pData; }; The Load() function of the WaveFile class loads the sound data and performs some minimal error checking. For example, I assume that only uncompressed, 16- bit WAVE files will be used. You can easily expand this class yourself if you need to load 8-bit files, etc. If a WaveFile object is created successfully and a WAVE file is loaded, the raw data can be accessed through the m_pData pointer. The following code shows the code for the Load() function of the WaveFile class: void WaveFile::Load(string filename) { ifstream in(filename.c_str(), ios::binary); //RIFF char ID[4];, 4); ease purchase PDF Split-Merge on to remove this watermark.
  5. 230 Character Animation with Direct3D if(ID[0] != 'R' || ID[1] != 'I' || ID[2] != 'F' || ID[3] != 'F') { //Error: 4 first bytes should say 'RIFF' } //RIFF Chunk Size long fileSize = 0;*)&fileSize, sizeof(long)); //The actual size of the file is 8 bytes larger fileSize += 8; //WAVE ID, 4); if(ID[0] != 'W' || ID[1] != 'A' || ID[2] != 'V' || ID[3] != 'E') { //Error: ID should be 'WAVE' } //Format Chunk ID, 4); if(ID[0] != 'f' || ID[1] != 'm' || ID[2] != 't' || ID[3] != ' ') { //Error: ID should be 'fmt ' } //Format Chunk Size long formatSize = 0;*)&formatSize, sizeof(long)); //Audio Format short audioFormat = 0;*)&audioFormat, sizeof(short)); if(audioFormat != 1) { //Error: Not uncompressed data! } //Num Channels*)&m_numChannels, sizeof(short)); //Sample Rate*)&m_sampleRate, sizeof(long)); Please purchase PDF Split-Merge on to remove this watermark.
  6. Chapter 10 Making Characters Talk 231 //Byte Rate long byteRate = 0;*)&byteRate, sizeof(long)); //Block Align short blockAlign = 0;*)&blockAlign, sizeof(short)); //Bits Per Sample*)&m_bitsPerSample, sizeof(short)); if(m_bitsPerSample != 16) { //Error: This class only supports 16-bit sound data } //Data Chunk ID, 4); if(ID[0] != 'd' || ID[1] != 'a' || ID[2] != 't' || ID[3] != 'a') { //Error: ID should be 'data' } //Data Chunk Size long dataSize;*)&dataSize, sizeof(long)); m_numSamples = dataSize / 2; //
  7. 232 Character Animation with Direct3D A UTOMATIC L IP -S YNCING In the previous section you learned how to load a simple WAVE file and how to ac- cess the raw PCM data. In this section I will create a simplified lip-syncing system by analyzing the amplitude of a voice sample [Simpson04]. The main point of this approach is not to create perfect lip-syncing but rather to make the lips move in a synchronized fashion as the voice line plays. So, for instance, when the voice sam- ple is silent, the mouth should be closed. The following function returns the aver- age amplitude of a voice sample between two points in time: short WAVE::GetAverageAmplitude(float startTime, float endTime) { if(m_pData == NULL) return 0; //Calculate start & end sample int startSample = (int)(m_sampleRate * startTime) * m_numChannels; int endSample = (int)(m_sampleRate * endTime) * m_numChannels; if(startSample >= endSample) return 0; //Calculate the average amplitude between start and end sample float c = 1.0f / (float)(endSample - startSample); float avg = 0.0f; for(int i=startSample; i
  8. Chapter 10 Making Characters Talk 233 void FaceController::Speak(WAVE &wave) { m_visemes.clear(); //Calculate which visemes to use from the WAVE file data float soundLength = wave.GetLength(); //Since the wave data oscillates around zero, //bring the max amplitude down to 30% for better results float maxAmp = wave.GetMaxAmplitude() * 0.3f; for(float i=0.0f; i
  9. 234 Character Animation with Direct3D Here I create a viseme for every 100 milliseconds, but you can try out different amounts of visemes per second. Of course the result will be a bit worse comparing this method to the previous one where the visemes were created manually, but the major upside with this approach is that you can quickly get “decent” looking lip- syncing with very little effort and no pre-processing. EXAMPLE 10.2 This example shows a simple lip-syncing system based on the amplitude of a voice sample. Play around with what visemes are assigned to which am- plitude range, the number of visemes per second, and perhaps the blending amounts. See if you can improve on this example and make the lip-syncing look better. C ONCLUSIONS This chapter covered the basics of lip-syncing and how to make a character “speak” a voice line. This is still a hot research topic that is constantly being improved upon. However, for games using thousands of voice lines, the focus is almost always on making the process as cheap and pain free as possible as long as the results are Please purchase PDF Split-Merge on to remove this watermark.
  10. Chapter 10 Making Characters Talk 235 “good enough.” In this chapter I showed one way of doing the lip-synching auto- matically using only the amplitude of a voice sample. Granted, this wouldn’t be considered high enough quality to work in a next-generation project, but at least it serves as a starting point for you to get started with analyzing voice samples. If you want to improve this system, I suggest you look into analyzing the voice data with Fourier Transforms and try to classify the different phonemes. F URTHER R EADING [Lander00] Lander, Jeff, “Read My Lips: Facial Animation Techniques.” Available online at animation_.php, 2000. [Simpson04] Simpson, Jake, “A Simple Real-Time Lip-Synching System.” Game Programming Gems 4, Charles River Media, 2004. [Lander00b] Lander, Jeff, “Flex Your Facial Muscles.” Available online at, 2000. ease purchase PDF Split-Merge on to remove this watermark.
  12. 11 Inverse Kinematics This chapter will introduce you to the concept of inverse kinematics (IK). The goal is to calculate the angles of a chain of bones so that the end bone reaches a certain point in space. IK was first used in the field of robotics to control robotic arms, etc. There are plenty of articles about IK in this field if you give it a search on Google. In this chapter, however, I’ll show you how to put this idea to work on your game character. IK can be used for many different things in games, such as placing hands on items in the game world, matching the feet of a character to the terrain, and much more. 237 ease purchase PDF Split-Merge on to remove this watermark.
  13. 238 Character Animation with Direct3D So why should you bother implementing inverse kinematics? Well, without it your character animations will look detached from the world, or “canned.” IK can be used together with your keyframed animations. An example of this is a character opening a door. You can use IK to “tweak” the door-opening animation so that the hand of the character always connects with the door handle even though the handle may be placed in different heights on different doors. This chapter presents two useful cases of IK. The first is a simple “Look-At” example, and the second is a Two-Joint “Reach” example. In short, this chapter covers the following: Inverse kinematics overview “Look-At” IK Two-Joint “Reach” IK A big thanks goes out to Henrik Enqvist at Remedy Entertainment for the sample code of the IK examples covered in this chapter. I NTRODUCTION TO I NVERSE K INEMATICS Before I cover inverse kinematics I’ll first cover the concept of forward kinematics (FK). Forward kinematics is something you’ve come across many times throughout this book already. Forward kinematics is the problem of solving the end point, given a chain of bones and their angles. An example of forward kinematics can be seen in Figure 11.1. FIGURE 11.1 Forward kinematics: Calculating the target, when knowing the origin, the bones, and their angles. Please purchase PDF Split-Merge on to remove this watermark.
  14. Chapter 11 Inverse Kinematics 239 The linked bones are also known as a kinematics chain, where the change in a bone’s orientation also affects the children of that bone (something with which you should already be familiar after implementing a skinned character using bone hierarchies). Forward kinematics come in very handy when trying to link something to a certain bone of a character. For example, imagine a medieval first-person shooter (FPS) in which you’re firing a bow. A cool effect would be to have the arrows “stick” to the enemy character if you have gotten a clean hit. The first problem you would need to solve is to determine which bone of the character was hit. A simple way to do this is to check which polygon of the character mesh was pierced and then see which bone(s) govern the three vertices of this polygon. After this you would need to calculate the position (and orientation) of the arrow in the bone space of the bone you’re about to link the arrow to. Next you would update the position of the arrow each frame using forward kinematics. Alternatively, the bone may have its own bounding volume, and then you can just check if the arrow intersects any of these instead. With forward kinematics you don’t know the end location of the last bone in the kinematics chain. With inverse kinematics, on the other hand, you do know the end location (or target) that you want the kinematics chain to reach. What you don’t know are the different angles (and orientations) of the joints (a and b in Figure 11.1). Solving the forward kinematics problem is relatively easy, but coming up with an efficient (and general) solution for the inverse kinematics problems is much harder (see Figure 11.2). FIGURE 11.2 Example of inverse kinematics. In Figure 11.2 you can see a 2D example of inverse kinematics. With inverse as opposed to forward kinematics, there is often more than one solution to a problem. In Figure 11.2, three example solutions to the same problem are shown (although there are more). Imagine then how many more solutions there are in 3D! Luckily when it comes to game programming, cutting corners is allowed (and often necessary). By reducing and adding more information about the ease purchase PDF Split-Merge on to remove this watermark.
  15. 240 Character Animation with Direct3D problem, you can go from this near-impossible problem to a quite manageable one. This chapter will cover some approaches to solving the problem of inverse kinematics for characters. S OLVING THE IK P ROBLEM Solutions to IK problems come in two flavors: analytical and numerical. With analytical solutions, you have an equation that can be solved directly. This is the fast and preferred way of solving IK problems. However, with several links in the IK chain, the analytical solution is rarely an option. Numerical solutions attempt to find approximate (but somewhat accurate) solutions to the problem. The approximations are usually done by either iterating over the result, which finally converges toward the solution, or dividing the problem into smaller, more manageable chunks and solving those separately. Numerical IK solutions also tend to be more expensive compared to their analytical counterpart. Two popular numerical methods for solving IK problems are cyclic coordinate decent (CCD) and the Jacobian matrix. Cyclic coordinate decent simplifies the problem by looking at each link separately. CCD starts at the leaf node and moves up the chain, trying to minimize the error between the end point and the goal. This approach can require a fair amount of passes over the IK chain before the result is acceptable. CCD also suffers from the problem of sometimes creating unnatural-looking solutions. For example, since the first link to be considered is the leaf (e.g., the wrist or ankle), it will first try to minimize the error on this link, which might result in really twisted hands or feet. The Jacobian matrix, on the other hand, describes the entire IK chain. Each column in the Jacobian matrix describes the change of the end point (approxi- mated linearly) as one of the links is rotated. Solving the Jacobian matrix is slow but produces better-looking results (in general) than the cyclic coordinate decent solution does. Since the Jacobian method is rather heavy on math, I’ll leave it out of this book, but for those interested, simply search for “The Jacobian IK” in your favorite search engine. L OOK -A T I NVERSE K INEMATICS To start you off with IK calculations, I’ll start with the simplest example: having only one bone orientation to calculate. Figure 11.3 shows an example of Look-At IK. Please purchase PDF Split-Merge on to remove this watermark.
  16. Chapter 11 Inverse Kinematics 241 FIGURE 11.3 Look-At Inverse Kinematics. In Figure 11.3 the character is facing the target (black ball) no matter where the ball is compared to the character. Since the target might be moving dynamically in the game, there is no way to make a keyframed animation to cover all possible “view angles.” In this case, the IK calculation is done on the head bone and can easily be blended together with normal keyframed animations. One more thing you need to consider is, of course, what should happen when the Look-At target is behind the character or outside the character’s field of view (FoV). The easiest solution is just to cap the head rotation to a certain view cone. A more advanced approach would be to play an animation that turns the character around to face the target and then use the Look-At IK to face the target. In either case you need to define the character’s field of view. Figure 11.4 shows an example FoV of 120 degrees. FIGURE 11.4 Limiting the field of view (FoV). ease purchase PDF Split-Merge on to remove this watermark.
  17. 242 Character Animation with Direct3D So what I’ll try to achieve in the next example is a character that can look at a certain target in its field of view (i.e., turn the character’s head bone dynamically). To do this I’ll use the InverseKinematics class. This class encapsulates all the IK calculations, the updating of the bone matrices, etc: class InverseKinematics { public: InverseKinematics(SkinnedMesh* pSkinnedMesh); void UpdateHeadIK(); void ApplyLookAtIK(D3DXVECTOR3 &lookAtTarget, float maxAngle); private: SkinnedMesh *m_pSkinnedMesh; Bone* m_pHeadBone; D3DXVECTOR3 m_headForward; }; The constructor of the InverseKinematics class takes a pointer to the skinned mesh you want to “operate on.” The constructor finds the head bone and does the necessary initializations for the IK class. The magic happens in the ApplyLookAtIK() function. As you can see, this function takes a Look-At target (in world space) and a max angle defining the view cone (FoV) of the character. Here’s the initialization code of the InverseKinematics class as found in the class constructor: InverseKinematics::InverseKinematics(SkinnedMesh* pSkinnedMesh) { m_pSkinnedMesh = pSkinnedMesh; // Find the head bone m_pHeadBone = (Bone*)m_pSkinnedMesh->GetBone("Head"); // Exit if there is no head bone if(m_pHeadBone != NULL) { // Calculate the local forward vector for the head bone // Remove translation from head matrix D3DXMATRIX headMatrix; headMatrix = m_pHeadBone->CombinedTransformationMatrix; headMatrix._41 = 0.0f; headMatrix._42 = 0.0f; Please purchase PDF Split-Merge on to remove this watermark.
  18. Chapter 11 Inverse Kinematics 243 headMatrix._43 = 0.0f; headMatrix._44 = 1.0f; D3DXMATRIX toHeadSpace; if(D3DXMatrixInverse(&toHeadSpace, NULL, &headMatrix) == NULL) return; // The model is looking toward -z in the content D3DXVECTOR4 vec; D3DXVec3Transform(&vec, &D3DXVECTOR3(0, 0, -1), &toHeadSpace); m_headForward = D3DXVECTOR3(vec.x, vec.y, vec.z); } } First I locate the head bone (named Head in the example mesh). Next I remove the transformation from the combined transformation matrix by setting element 41, 42, 43, and 44 in the matrix to 0, 0, 0, and 1 respectively. I then calculate the in- verse of the resulting matrix. This lets you calculate the head forward vector (in the local head bone space) shown in Figure 11.5. FIGURE 11.5 The forward vector of the head bone. ease purchase PDF Split-Merge on to remove this watermark.
  19. 244 Character Animation with Direct3D The forward vector of the head bone is calculated when the character is in the reference pose and the character is facing in the negative Z direction. You’ll need this vector later on when you update the Look-At IK. Next is the ApplyLookAtIK() function: void InverseKinematics::ApplyLookAtIK(D3DXVECTOR3 &lookAtTarget, float maxAngle) { // Start by transforming to local space D3DXMATRIX mtxToLocal; D3DXMatrixInverse(&mtxToLocal, NULL, &m_pHeadBone->CombinedTransformationMatrix); D3DXVECTOR3 localLookAt; D3DXVec3TransformCoord(&localLookAt, &lookAtTarget, &mtxToLocal ); // Normalize local look at target D3DXVec3Normalize(&localLookAt, &localLookAt); // Get rotation axis and angle D3DXVECTOR3 localRotationAxis; D3DXVec3Cross(&localRotationAxis, &m_headForward, &localLookAt); D3DXVec3Normalize(&localRotationAxis, &localRotationAxis); float localAngle = acosf(D3DXVec3Dot(&m_headForward, &localLookAt)); // Limit angle localAngle = min( localAngle, maxAngle ); // Apply the transformation to the bone D3DXMATRIX rotation; D3DXMatrixRotationAxis(&rotation, &localRotationAxis, localAngle); m_pHeadBone->CombinedTransformationMatrix = rotation * m_pHeadBone->CombinedTransformationMatrix; // Update changes to child bones if(m_pHeadBone->pFrameFirstChild) { m_pSkinnedMesh->UpdateMatrices( (Bone*)m_pHeadBone->pFrameFirstChild, &m_pHeadBone->CombinedTransformationMatrix); } } Please purchase PDF Split-Merge on to remove this watermark.
  20. Chapter 11 Inverse Kinematics 245 This function uses the shortest arc algorithm [Melax00], to calculate the angle to rotate the head bone so that it faces the Look-At target. Figure 11.6 shows the shortest arc algorithm in action. FIGURE 11.6 The shortest arc algorithm, with X being the rotation angle you need to calculate. The head forward vector is calculated in the initialization of the Inverse- Kinematics class. The target you already know; all you need to do is calculate the normalized vector to the target in bone space (since the head forward vector is in bone space). Calculate the cross product of these two vectors (head forward and target vector) and use that as the rotation axis. Then the angle is calculated and cap’d to the max rotation angle and used to create the new rotation matrix (this is the matrix that will turn the head to face the target). Finally update the combined transformation matrix with the new rotation matrix and be sure to update any child bones of the head bone as well using the SkinnedMesh::UpdateMatrices() function. ease purchase PDF Split-Merge on to remove this watermark.
Đồng bộ tài khoản